The GNU extension language - <AUTHOR> http://wingolog.org/archives/2011/08/30/the-gnu-extension-language <comment><USER2> When it comes to extensibility, Emacs and Emacs Lisp have one of the most compelling stories of all time. (In fact, is there anything that beats them?) If Andy is right that Guile can do for GNU what Elisp did for Emacs, only better, it would be downright amazing.

I wonder, though, whether some things that made Emacs/Elisp so successful would get lost in the switchover. For example, while dynamic scoping is bad for writing robust complex software, it does make Emacs easier to extend.

Emacs is not really an application or a platform, it is a framework: a framework for a text editor, one that comes with a well-thought-through model and a flagship application (Emacs itself) that doubles as a development environment. I don't mean "model" as in MVC, but in the sense of a consistent set of concepts that can be composed to do things. In Emacs those concepts are things like buffers and windows and text properties. Emacs has one of the best models I've ever seen; it should be taught as an example of how to do this kind of design. Emacs on the surface may be clunky and unlovable, but dive underneath and its world is beautiful.

When people extend Emacs, they're building programs out of that model using tools designed for building programs out of the model (which themselves were built out of the model). If you shift from Emacs to GNU as a whole, you're moving from that single -- self-consistent and composable, not to mention interactive -- design context into a hugely diverse environment with no common design context at all. If Emacs is a planet, GNU is a universe. It's not clear how well Emacs' extensibility magic can be reproduced at that level. If the benefits come mostly from the language, Andy may be right. I hope he is. But if the benefits come mostly from Emacs' framework and interactive environment (both of which are inseparably bound up with the language), then the magic is likely to be left behind with them.

Still, it's hard to argue against his case for choosing the most powerful possible extension language so as to maximize the extending that people will be able to do with it. To argue against this in favor of familiarity is particularly to miss the point.

<comment><USER3> How does dynamic scoping make Emacs easier to extend?

<comment><USER2> _How does dynamic scoping make Emacs easier to extend?_

It gives you easy access to everything that's going on at runtime. For example, many Emacs buffers are read-only or have read-only bits of text (e.g. the prompt in a Slime REPL). If your code tries to change that text you get an error. But there's a trivial idiom for getting around it:

      (let ((inhibit-read-only t))
        (do-the-thing-that-changes-text))

Any code invoked during the execution of this form, such as the system code that enforces read-onlyness, will now behave differently.

Because Emacs is dynamically scoped, everything works this way. You don't depend on the author of some module having said in the past, "Someone might want to modify this variable at runtime someday, so I'll declare it to be dynamic." Everything is readable and writeable by default. This is a big deal for Emacs' extensibility.

Dynamic scoping isn't good in general, but Emacs has benefited handsomely by making this tradeoff in its own case. There are thousands of useful Elisp modules that interoperate with remarkably little fuss, all extending the same system. Yet the language is dynamically scoped and has zero encapsulation. Most people, I think, would deny that that was possible if it didn't exist. Many people still deny it is possible even though it does!

<comment><USER3> I don't think dynamic scoping makes it possible -- I think it is used simply because it is there. If it wasn't there, some other solution, just as good (if not better) would be used.

When writing e-lisp, I find dynamic scoping limiting, and using lexical-let, especially when I want curried/partial-application, is extremely annoying.

For me, extending emacs is harder because of dynamic scoping. It is entirely possible to have code be executed in some sort of "Reader" monad/env, and execute it in a different than default env -- which gives you "dynamic scoping" on whatever input you want, without contaminating every other name.

I think almost every negative feature you could possibly implement would find "good use" that defended its existence (Dynamic scoping, $_ in Perl, implementation inheritance, etc).

</comment></comment></comment></comment><comment><USER4> Although treasonous to say, I don't think a sexp-based language is appropriate for a wide community of programmers. (I'm currently enjoying coming back to PLT Racket from Clojure, so please don't write me off as someone who can't get over the initial shock.)

Macros make it _too_ easy to define new syntax, to the point where things that should be core language features are neglected, only to be continually re- half-implemented, etc. Take a look at what passes for 'good enough' for destructuring in Clojure, both the older let-based and the newer core.match (which still lacks the extensibility of Racket's match). The deficiencies can always be fixed 'later', right?

No syntactic difference between function application and special forms means that one requires an editor that can somehow differentiate (slime/drracket approaches) to even have a _shot_ at quickly scanning unfamiliar code (say, quickly looking at what an implementation actually does). It's pointed out that Guile will handle other languages (imho this is tacitly admitting uncertainty about Scheme). But multiple languages only stand to make the code- reading problem even worse! Never mind that Scheme will be the only first- class language (other languages can't consume macros) so alternatives are at best advertisements to get people hooked and having to learn the 'real language'.

The little bit of Ruby I've written opened my eyes to how much you can do with just first-class code blocks, and made me understand why something like SLIB would opt for explicit lambdas everywhere instead of macros. Macros are so enticing to cut down on boilerplate, but they're visually indistinguishable, and without plenty of forethought they're not composable.

Clearly many skin syntaxes for Lisps have been attempted and never caught on. I don't currently have much to add on that front besides a suggestion of not being afraid of well thought out concrete syntax (how many base special forms are there, really?). But really, I think my whole post shows the main problem with Lisp, at least for me. It's so easy to do things and it's such a blank slate that no matter what I sit down to do, I always end up trying to create a new language.

<comment><USER5> _Although treasonous to say, I don't think a sexp-based language will ever be adopted by a wide community of programmers._

The problem with this argument is that a sexp-based configuration language is already adopted by a community of programmers: Emacs Lisp. Emacs itself ships with over a million lines of Lisp code, and there are thousands of extensions available from around the Internet.

The problems you worry about with respect to macros have simply not happened; most people name macros the same way (starting with _def_ or _define-_ ), and most macros are written "cleanly" in such a way as to not surprise the programmer. (I'll be honest: I can't think of one time I've ever hit a but in a Lisp program caused by a strange macro.)

Emacs Lisp shows that if you give someone a turing-complete programming language on top of a powerful piece of software with a clean and well- documented API, programmers will ignore the warts and enhance the software in ways you never thought possible.

Guile is not a perfect programming langauge, and I would never use it by choice. But if I started using some piece of software that used it for configuration, I know I would love it because it would stay out of the way and _let me make things_. Once you are making cool things, you don't care about the syntax or programming language features at your disposal: you just make it work.

Some other examples of this include JavaScript for Firefox extensions and Vimscript for Vpim extensions. JavaScript is very ... minimal compared to other languages like Perl, Python, Ruby, Haskell, and so on, but people write great browser extensions anyway. Vimscript is the weirdest thing I've ever seen but Vim is able to do a lot of cool stuff despite this.

The only thing a successful extension language needs is something worth extending. If you use Guile as your extension language, you can concentrate doing that instead of writing your own programming langauge toolchain.

<comment><USER2> _I can't think of one time I've ever hit a bu[g] in a Lisp program caused by a strange macro._

What about macros that have the same name? In Emacs, the convention is to name your functions foo-this and foo-that if your program is foo.el. But this does not seem to be true for macros. Is that because macros don't leap outside the compilation context of foo.el? (Edit: nope, I checked it and they do.) Or is it that people are sloppier about naming their macros?

Edit: or is it rather that, if your foo.el defines a macro 'blah, and I write bar.el that says (require 'foo) and then defines its own macro 'blah, it's my macro that gets used to compile my code, so the one in foo.el can't do any harm to it?

<comment><USER5> The naming convention for macros is a little weird. My rule is: if the macro is user-visible (and not something to make your module easier to implement), it should start with define- or with-, and should include a uniquifier somewhere in the name. If you were writing an extension called foo, you might provide a macro called with-current-foo or define-foo-bar.

If someone already has a macro with this name, then don't call it that.

The reason for ignoring the normal package-function naming convention is that it's simply ugly to write code like:

       (eproject-with-current-project "foo" (whatever))

when you could write

       (with-current-project "foo" (whatever))

<comment><USER2> I get (and practice) the part about user-visible macros. But what about macros that _are_ just to make my module easier to implement? I often write those to concisify (well, you did say "uniquifier") my code. There's a tension here: encoding the package name in the macro name can easily make the name so long that the whole thing is why-bother, but using a short name runs the risk of a collision. For example, paredit.el defines XCOND. Although I don't have an XCOND, that's just the sort of macro name I might have chosen.

</comment></comment></comment></comment></comment><comment><USER6> This quote from the comments summarizes my thoughts on this:

" _Robert says:

30 August 2011 11:55 PM

It isn't whether Guile is good, bad, usable, or not usable. It is whether people will and do USE it. I have not seen that as the case for Guile or Scheme in general._"

I don't know anybody anywhere who has even tried to use Guile. I don't even know if anybody I know have ever looked at it. I don't know how many people even know it exists.

I personally love Scheme as a language, but I honestly can't see myself ever investing enough time into Guile to be able to write program extensions in it. I don't see my friends (some of whom I believe would write extensions to popular open source software if they could do so in a language that is sufficiently similar to one they already know - I certainly would be a lot more likely to do so and I'm not even Lisp averse) doing so either.

" _First of all, what does it mean to have a GNU system based on a non-GNU technology? What would be left of GNU?_ "

So pick a non-GNU language and create a GNU implementation.

Personally, I think the best choice would be Lua. Its mature, a lot of people already use it as an extension language and its very similar to a lot of popular languages, so resistance for people to learn it is low (it is reasonably similar to Javascript and to a lesser extent, Ruby and Python; also, as has been proven by the game industry, it is easy to learn for C, C++ and Java programmers. Finally, while not GNU, it has some exceptionally good implementations).

<comment><USER7> Well, I've never used Guile, but I _have_ used Elisp, to great success. It allowed me--with little previous Elisp experience--to easily write a minor and a major mode without having to spend too much time in the manual. Given that Guile is very similar--just more modern and polished--to Elisp, I definitely see it as a reasonable extension language.

I think the issue with the limited number of extensions written in Guile is more with the projects that can be extended than with the language--I'm not aware of any that I use, and certainly none that I would like to extend.

<comment><USER6> I get the impression that the majority of emacs users are not averse to learning seemingly weird and dificult new things (otherwise you _probably_ would not have learned to use emacs in the first place). The majority of users are, in my experience anyway, put off by all sorts of things (and often quite unrelated to the merits (or lack of) the language itself). I have also experienced quite a bit of lisp hostility. All of these things add up for me to feel there are better choices as an extension language if you want the _average_ [1] user to write extensions.

[1] by average I mean that they have the technical skill to do so, but aren't terribly invested in learning lots of new things. I imagine (I'm a vim user myself and I find its true for me) that an emacs user has already invested in the tool enough that learning a little more to make it even more useful is not a big deal.

<comment><USER7> I can't help thinking that your average user is much less likely to write an extension at all. Also, I suspect a disproportionate number of GNU program users sufficiently invested in them to write extensions are either Emacs users already (it is Gnu Emacs, after all) or at least fairly familiar with Lisp.

The sort of person who doesn't go out of his way to learn a new tool is also the sort of person who probably doesn't go out of his way to _extend_ an existing tool either. I think targeting advanced users for writing extensions is completely reasonable.

Plenty of "average" (by your definition) programmers who use Eclipse know Java well enough to write Eclispe plugins. How many of them do? I suspect a large portion of those who do would have written the plugins in Guile as well.

<comment><USER6> Probably not all that relevant here, but I think that Java's verbosity is a barrier to entry in itself, even if Java programmers do not think so. When I worked in a Java shop a few years ago I would have happily coded up quick extensions and scripts[1] in something like python, but never even considered doing so in Java because there was too much effort involved. I guess I'm arguing that the users must feel that the benefit of writing a script or extension must feel like its much greater than the effort of not doing so _and_ the effort of writing the extension and that Guile is just plain too unfamiliar to most that the effort _seems_ to outweigh the benefits, even if in reality this is not true.

[1] I did actually write a few Python scripts to generate repetitive Java code because I felt that doing so would be much less effort than writing the code by hand.

</comment></comment></comment></comment></comment><comment><USER8> Quote: " Scheme's macros have no comparison in any other language."

I argue that Erlang parse transforms are at least as powerful. Interesting that he mentions Erlang, but doesn't mention that! (Erlang probably isn't ideal as an /extension/ language for other reasons, though)

<comment><USER9> erlangs parse transforms are indeed just as powerful but no where near as easy to use. They allow you to modify your source file in pretty much the same way but the ast you are modifying is way more complicated than lisps and much more difficult to properly modify.

If Lisps macros are difficult for the average user to learn then Erlangs Parse Transforms are even more so and a much larger barrier to usage than lisps. It's rare for an erlang developer to reach for a Parse Transform it's rare for a lisp developer not to reach for one.</comment></comment>


Tell HN: Hacker News India - <AUTHOR>

Hi,
We just launched Hacker News India at http://hackernews.in
Hacker News is awesome but there are lots of local issues that Indian hackers need to discuss (such as local events, infrastructure issues etc) that don't get good visibility on HN. We have started HNI to fix that.
It's a not for profit initiative. Please checkout the site and contribute to it (by commenting or posting questions).
Any suggestions are welcome. Would be lovely if they are posted there :) <comment><USER2> Changing the name is the best course of action. The name 'hackers news' may not be copyrighted, but that does not make using the exact same name the right thing to do.

The name 'hacker news india' may deceive at least a few people into believing that the site is somehow affiliated to the original site.

Furthermore, I don't think that using the same name will somehow make the site in par with the original hacker news. It will be the quality of the discussions that will decide the fate of the site.

I believe that the same name was used with the best of intentions, but now that the mistake is being pointed out, it is better to change the name.

<comment><USER3> I would agree, especially given that the site looks exactly like Hacker News with different colours. Quite a few of the articles up there look like they'd be fine here.

</comment></comment><comment><USER4> Bad Idea. I am Indian and I plan to stay away from this.

"Hacker News" is a name used by PG for his webapp and you shouldn't use it (or variants) for yourself without getting explicit permission from him (You didn't did you?).

Imagine setting up a "Google India" or "Stanford India" site and claiming "if they want to operate these sites they can, we are not making any money off this". If those are shady, so is this.

If you just want an HN like forum, why not use the open source arc code but make your own name?

While I am not convinced there are issues that are exclusive enough to "Indian hackers" (shudder) that need a separate forum, others may have such a need. Good for them.

But please don't steal the name of a successful site without explicit permission from the owner.

Overall an amateurish move, if not illegal. (All purely imo - my fellow "Indian hackers" can downvote)

My response = flip bit. Move On.

<comment><USER5> I agree with the idea, but disagree with the choice of name.

</comment></comment><comment><USER6> Are you sure that you can just starting using this name?

<comment><USER7> This name is owned by paul graham dude. And I don't think he will come in the way of gathering a few hackers on a news site.

<comment><AUTHOR> Also, we are not making money off this and if YC wants to run the place, we are more than happy to let them :)

</comment></comment></comment><comment><USER8> The fact that PG released HN code as open source means that he wants more people to use it. The issue about the name is correct, we shld think of a different name for HN India - so that there is no false impression of any affiliation with HN.

But I think a local flavor is surely a good idea and more counties / geographies should create their versions. Just in 3-4 days the Indian HN has gotten a lot of traction.

</comment><comment><USER9> enough comments on NAME, hope they got it. I wish they could get someother useful suggestions.

</comment><comment><USER10> What different things you are going to do with HNI ? I mean I wanted to know the purpose of creating whole new thing similar to HN. Also need to manage both accounts personally I don't like many accounts.

<comment><USER7> I get the point the op is trying to make there. You see I was longing for a lot of time for something similar but which can also discuss local issues. I just feel left out when there is discussion only about the startup scene in silicon valley SF etc.

I would strongly support this initiative.

<comment><USER3> If you don't like the silicon valley skew then post more non-silicon valley stuff here.

There's a certain valley skew here because of the people engaged by HN, YC and PG, but it's hardly exclusive.

<comment><USER7> It becomes exclusive when most of you are from silicon valley. Also if I were to post something like this[1] here I wouldn't get many answers.

<http://hackernews.in/item?id=112>

<comment><USER3> Your comment would carry more weight if I were from silicon valley (sorry, UK here).

As for answers, post according to your target market. Surely if you post while fellow Indians are awake you'll get answers from them? How many users are on hackernews.in that aren't on news.ycombinator.com?

</comment></comment></comment></comment></comment><comment><USER7> Where is the ask and jobs sections? I love the first and long for an indian edition of the latter.</comment>


Git from the inside out - <AUTHOR> https://codewords.recurse.com/issues/two/git-from-the-inside-out <comment><USER2> Certain systems can best be understood as black boxes. You put some commands in and magic happens. Git was not designed to be such a system and early users of git know this.

During the last 5 years, many GUIs have filled in this gap, making it increasingly likely to find people completely stuck because they miss knowledge of the foundations.

Git is a utility to manage an append-only repository of tree-objects, blobs and commits. To help humans, git adds

\- human-readable pointers (branches, HEAD, stash)

\- an method to incrementally add changes (staging/index/working area)

\- a method to append tree-objects, blobs and commits from repository to another

\- some commands which alleviate steps in common tasks

These last set of commands cause pain, as users without foundational knowledge, do not realize these commands are compounding many small steps.

<comment><USER3> > During the last 5 years, many GUIs have filled in this gap, making it > increasingly likely to find people completely stuck because they miss > knowledge of the foundations.

Arguably, these GUIs became popular because Git by itself offers awful UX. I am all for obscure commands and switches (I use Linux and prefer CLI to mouse), but Git really took it to the next level. And it's not as if it couldn't be done better (as Mercurial shows).

As a user I don't want to know the intrinsic details about how some system is implemented. I just don't care - give me an external model which helps me use it, and leave it at that. I have my own code I need to worry about.

If most users get the "wrong" mental model when using Git, then the problem lies with Git, not with users. </rant>

<comment><USER4> A large part of the point of Git (in contrast to earlier tools like CVS) is that the internal data model that it enforces is one that _works_ in the long term in distributed project settings.

Distributed source control requires a kind of _hygiene_ , much like dental hygiene, or like cryptographic security. In the practice of hygiene, what's _convenient_ is usually opposed to what's _sustainable_. Hygiene is effectively the set of discoveries of _un-natural_ or _non-intuitive_ ways to do things that _work better_ than the equivalent intuitive/natural practices. Thus, any process that is "hygenic" is going to cause at least a little bit of pain or annoyance to follow—if it didn't, it'd be "natural."

And _that 's_ the fundamental problem with Git: that it tries to paper over the fact that it's "hygenic" by presenting itself as something seemingly "natural." People have the wrong mental model of Git because Git tries to meet them in the middle, by translating its (correct, sustainable) internal model into (broken, leaky) familiar abstractions.

Git isn't your friend. Git is your toothbrush. Imagine your dental hygienist telling you they're disappointed in how infrequently and badly you're brushing your teeth. The right response to that isn't _blaming the hygienist for making toothbrushes annoying to use_ , right? It's _shame_ , shame that you haven't bothered to overcome the stupid stubborn ignorance getting in the way of you taking better care of your teeth. Shame that you can't "meet your toothbrush where it lives."

Meet Git where it lives.

<comment><USER5> If git exposed a consistent, understandable interface to its data model you'd have a point.

Mercurial exposes a much more consistent, much more understandable interface to a very-similar-to-git data model, so it's not like this is impossible.

So it'd be really nice if people could be allowed to point out the shortcomings of git's command-line interface without other people implying their complaints must be rooted in being too dumb and/or too lazy to understand git.

<comment><USER6> Having never used mercurial. What are some of the places where mercurial provides a better interface than git?

<comment><USER7> Just read this: [http://stevelosh.com/blog/2013/04/git- koans/](http://stevelosh.com/blog/2013/04/git-koans/)

And then think about that Mercurial simply makes handling branches/updates/resets in a very clean and consistent way.

<comment><USER8> Clearly not a new post, but the first time I had seen it, clearly spells out what has frustrated me (and many others) for a very long time.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> See also "Git from the Bottom Up" [https://jwiegley.github.io/git-from-the- bottom-up/](https://jwiegley.github.io/git-from-the-bottom-up/)

I read lots of tutorials on Git when I started with it just a few years ago, and that's the one that best helped me grok it.

<comment><USER10> Am I the only one who read the "official" Git ebook and found that perfectly understandable?

[https://git-scm.com/book/en/v2](https://git-scm.com/book/en/v2)

<comment><USER9> Yes, the book is excellent and also recommended :) But for me, having an understanding of git's internals really helped provide a foundation for the rest. Otherwise a lot of the stuff (like rebases, or amend commits...) were just black magic.

</comment></comment></comment><comment><USER11> I like the writing style and the scope of the piece. Well done.

I kind of wish there were at least mentions of git plumbing commands where appropriate, to shake off one more level (half a level?) of magic. For example, just link to some information on `git hash-object` in the section on `git add`. Footnotes would probably be enough. No need to bog down the relatively quick pacing. Sometimes it can be hard to discover which plumbing commands correspond to the actions mentioned.

Most git tutorials come with diagrams of blobs and trees and branches with all the arrows and color coding. They get the meaning across but often seem to come with a bit of a mental disconnect from what is actually happening in the working directory and .git directory. Does anyone know of a tool to display that kind of diagram in real time while you are making commits or checking out new branches? It could bring an extra level of interactivity to the presentation. Imagine if the graphs on this page were updating live while you had to type the git commands to get them to update AND you could monitor the filesystem at the same time, showing exactly which files were changed by the command.

<comment><USER12> Github built one: [https://try.github.io](https://try.github.io)

More: [http://sixrevisions.com/git/interactive-git- tutorials/](http://sixrevisions.com/git/interactive-git-tutorials/)

</comment></comment><comment><USER13> [http://learngitbranching.js.org/?NODEMO](http://learngitbranching.js.org/?NODEMO)

I haven't seen a git article link to this amazing website recently. By far one of the best ways to teach someone git is to walk someone through git by executing commands and allowing them to see the visual representation of those commands.

There is also an amazing single-player learning mode.

</comment><comment><USER14> OT (or not) -

Of 20 most popular Qs on SO, 7 now ask how to do trivial operations in git.

[http://data.stackexchange.com/stackoverflow/query/36656/most...](http://data.stackexchange.com/stackoverflow/query/36656/most- upvoted-answers)

Take it or live it, git has facts-based proven track record of ui wtfness.

<comment><USER12> The two top questions are "How to modify existing, unpushed commits?" and "How to undo last commit(s) in Git?". That could be interpreted as evidence in favor of a staging area. People actually need to change committed things very often.

On the other hand, on place 20 is "How to undo 'git add' before commit?", which displays a UI shortcoming with staging.

<comment><USER14> It suggests staging is confusing for newcomers (or maybe - for the mass in general).

<comment> crooked-v A simple way to make it clearer: have 'git stage <file>' and 'git unstage <FILE>' commands, instead of the extraordinarily unintuitive 'git checkout -- <file>' for the latter.

</comment></comment></comment></comment><comment><USER15> git by itself, I recommend reading the README on the initial commit:

[https://github.com/git/git/commit/e83c5163316f89bfbde7d9ab23...](https://github.com/git/git/commit/e83c5163316f89bfbde7d9ab23ca2e25604af290)

It was only 1000 lines of C at the time, so it couldn't have needed a million different articles and blogs to explain.

That doesn't give you a workflow or explain any of the more advanced features. The workflow you can get from any cookbook list of shell commands, and the advanced features you can get from the manual.

<comment><USER16> Direct link to the README: [https://github.com/git/git/blob/e83c5163316f89bfbde7d9ab23ca...](https://github.com/git/git/blob/e83c5163316f89bfbde7d9ab23ca2e25604af290/README)

It's interesting to note that what is currently the index/staging area was, back then, only a cache. (which is also why you'll still find it called any of these three names, participating in the overall confusion)

</comment></comment><comment><USER17> Off-topic: this must be one of the most beautifully designed sites I've come across lately.

I wish more sites were an oasis of calm like this.

</comment><comment><USER18> Very good read. I was just hacking around with a tiny Python program that implemented enough to init, add, commit, and push itself to GitHub. It's all very simple until you get to the index format ... which isn't that bad, but it's definitely more complicated than this article makes out.

She refers to .git/index as a text file where "each line of the file maps a tracked file to the hash of its content". However, .git/index is actually a binary file where each entry is a bunch of different fields like creation time and modify time and SHA-1 encoded in binary. See [https://github.com/git/git/blob/master/Documentation/technic...](https://github.com/git/git/blob/master/Documentation/technical/index- format.txt)

So I wasn't sure whether this part of the article was simply wrong, or whether git index format "version 1" was text, or something else?

<comment><USER18> For future reference: I asked Mary (the author) about this and she said, "Great point! As far as structure of the data, the file may as well be a text file of lines of information. I decided to omit talk about how git actually stores data to focus on the structural concepts. Maybe I should have included a bullet point. I don't actually control the version of the article that was posted on HN. but I'll certainly update the version on my website."

</comment></comment><comment><USER19> See also [http://eagain.net/articles/git-for-computer- scientists/](http://eagain.net/articles/git-for-computer-scientists/)

</comment><comment><USER20> I think this is a good way to teach git.

I approach the whole thing similarly during my trainings and wrote a few dirty scripts to generate an image of what the repository looks like using graphviz [https://gist.github.com/nibrahim/6119925](https://gist.github.com/nibrahim/6119925)

<comment><USER21> I did similar with mermaid:

[https://github.com/ianmiell/learn-git-the-hard- way](https://github.com/ianmiell/learn-git-the-hard-way)

[https://github.com/ianmiell/learn-git-the-hard- way/blob/mast...](https://github.com/ianmiell/learn-git-the-hard- way/blob/master/learngitthehardway.pdf)

</comment></comment><comment><USER22> This is how I teach git when I do training sessions for companies. I really dig this approach; the plumbing on top of Git is not really sufficiently abstracted to avoid knowing this stuff, but at the same time it tries to hide just enough of it to end up biting you when (not if) something goes sideways.

</comment><comment><USER23> This reminds me of The Git Parable which helped me understand git when I was first getting started. I really do believe that you have to understand how git actually works under the UI in order to have long term success using it. Whether it's bad or good to need this understanding can be debated but I believe that it is truth.

[http://tom.preston-werner.com/2009/05/19/the-git- parable.htm...](http://tom.preston-werner.com/2009/05/19/the-git-parable.html)

</comment><comment><USER24> The thing that is missing from all tutorials is that branches, tags and everything else are just pointers to a node in the tree. I.e. branches have no "content".

</comment><comment><USER25> Good stuff. Can't help but agree with others that I don't really want to be forced to be intimate with Git at a gory insides level, though. I use tortoise Git, and perhaps it's removed the temptation to experiment with things that could blow my foot off, but 99% of the time I have no reason to drop to the command line--nor do I want to.

</comment><comment><USER26> There's also this from the same author: [https://maryrosecook.com/blog/post/git-in-six-hundred- words](https://maryrosecook.com/blog/post/git-in-six-hundred-words)

She's a really clear informative writer

</comment><comment><USER27> This is all simple stuff. It's when after a rebase there are conflicts in files where they shouldn't be and the CI says "failed" then my knees weaken and i yearn for a guide that would explain everything

</comment><comment><USER28> Explaining version control might be a bit challenging. Distributed version control is a bit more challenging. So teaching git can be a lot to take for a newcomer.

I went from subversion to git. In retrospective, subversion was much simpler conceptually (but problems like syncing branches were harder).

I found myself once explaining a git concept based on the plot of Back to the Future II. I think it was a perfect example to how to resolve some merge problem.

There are some "git cheatsheets" that provide a very straightforward graphical explanation of what some commands do. That helped me to consolidate some concepts.

</comment><comment><USER29> Anyone really interested in git internals should look at git-draw.

[https://github.com/sensorflo/git-draw/wiki](https://github.com/sensorflo/git- draw/wiki)

</comment><comment><USER30> Worth linking to the excellent YouTube version of this too: [https://www.youtube.com/watch?v=fCtZWGhQBvo](https://www.youtube.com/watch?v=fCtZWGhQBvo)

</comment><comment><USER31> This is really excellent. I have now read it three times, running through the commands as I go, and for the first time I feel like I actually understand git.

Here are some commands I found useful as I went through the tutorial: To show a given commit without diffs but with tree and parent sha1s: git cat-file -p sha-1

To show a given commit including diffs: git show sha-1

To show a given tree (blobs and subtrees): git ls-tree sha-1

Show the tree recursively (i.e. show sha1 of all files, ignoring folder) git ls-tree -r sha-1

</comment><comment><USER32> This is one of the best ways to lean new techonology. I remember that long back ago when i was in college i had some trouble understanding some aspects of web servers and so i decided to write a small web server in perl and soon soon i kinda knew it inside out. Same for writing my own Smtp client, the knowledge i gained from it will be with me forever.

<comment><USER33> True. Never understood git restriction to empty folders until I had to build my filesystem-on-database

</comment></comment><comment><USER34> This kind of makes me wonder why you have separate objects/ stores for each repository you have checked out - it seems like you could have just one in your $HOME that all your git repositories share.

</comment><comment><USER35> Read and upvoted this previous time it was posted. Reread and upvoted it this time too.

This is a good example explaining why reposts should be not only allowed, but encouraged on HN.

</comment><comment><USER36> Or just check out the talk with the kindergarden teacher wearing bib pants that stacks balls and pins... Git for ages four and up I think.

</comment><comment><USER37> Author has a lot of good reads on her blog. Also, some of her live coding demos on YouTube are fun to watch.

</comment><comment><USER38> This is how you do git. Read it. Learn it. And you will see the light of version control awesomeness!

</comment><comment><USER39> There are a lot of comments here such as "Git has a horrible UI/UX", "I don't want to have to learn the inner workings of a tool", "Certain systems are designed so well that they can be understood as black boxes", and a lot of other complaining about Git and it's idiosyncrasies.

I think this needs to be dissected a bit. First, Git operates in a manner (internally) that is foreign to most users of _other_ SCM systems. Second, Git has a bit of a "tacked-on" nature to it's CLI that can make use cumbersome for newcomers especially when they have been taught the shortcuts before the fundamentals.

For the first problem, I think this is where the black-box comments apply. And honestly, I think treating SCMs as black boxes is what got us into the situation we were in before Git. Version control, branching, merging, change management, and change deconfliction are _hard_ problems, IMO. Personally, I think the base level functions that Git provides, combined with Git workflows from Atlassian (and others) really helps provide a daily routine to handle these situations. After cloning: branch -> change -> index (or update) -> commit -> pull -> fix conflicts -> commit -> push -> merge (or pull request) -> repeat. There are some variations depending on your branching model, but by-in-large this is what prevents regressions and forces people doing the committing to resolve the changes and not to break master.

I think you need to understand the "internals" of any SCM to really be able to conquer the challenges of distributed version control and the complexities of modern software development. I've worked in Rational ClearCase shops, and we needed a ClearCase guru on site too. Every team should have a Git guru.

For the second problem, yes, the CLI is a bit clunky at times. This, combined with a misunderstanding of Git fundamentals can lead you down some bad paths. Cleaning up the CLI is an independent problem from Git internals - and I'll admit some taxonomy/hierarchy/ontology/whatever of commands is probably needed to refine the day to day workflows. However, if you mess up the repo because you don't understand the branching and merging model, you are going to have to use the more "specialized" commands which, let's face it are going to be a bit more cryptic. This is the same for any system that has some maintenance or repair type functionality.

This is why I say, learning how Git works, allows you to learn the branching model better, which will hopefully allow you to avoid those particularly thorny paths.

Sure, you can choose some other SCM system that seems less cryptic or easier to use, but you will likely find yourself in a bind someday in those systems that you will need it's cryptic commands to get out of. Or more likely, doing a lot of work that Git would have allowed you to do in a fraction of the time.

</comment><comment><USER40> Too bad git clone and fetch are still not resumable.

<comment><USER41> how would you implement it?

</comment></comment><comment><USER42> Mods, can you please add 2015 to the title?

</comment><comment><USER43> just when I needed the most. Thanks a lot homie!</comment>


How to lead by first principles - <AUTHOR> https://medium.com/@mmccue/the-most-powerful-lesson-i-ve-ever-learned-in-business-4d89e95ab250 <comment><USER2> Original posting got 30+ comments:

[https://news.ycombinator.com/item?id=11154125](https://news.ycombinator.com/item?id=11154125)</comment>


Ask HN: Any value in HTTP/2 for small sites with local users? - <AUTHOR> I am contemplating moving a few of my informational sites to HTTP/2 since the more efficient networking model appeals to me.  Because of my current setup this would require some effort to do properly.
I have proxied an existing tiny fossil site of mine behind Cloudflare to try http HTTP/1.1 and https HTTP/2 access side by side and cannot see much overall difference from local access to my existing http HTTP/1.1 sites.  I am using tools such as WebPageTest.org.
My sites are simple and low traffic and non-contentious (no strong need for https), and I already make an effort to keep them small and efficient.
Should I bother? <comment><AUTHOR> I've since been working on this:

[http://www.earth.org.uk/note-on-carbon-cost-of- CDN.html](http://www.earth.org.uk/note-on-carbon-cost-of-CDN.html)

I can see no real gain for small lightweight pages with users close by geographically, but heavier pages with >=10 objects do see a benefit IMHO, up from ~1--2 objects per connection to many times that.

</comment><comment><USER2> If you're already doing things like bundling any javascript, image sprites etc etc, you're probably not going to 'see' much benefit. Hence the tools aren't showing much difference.

The big benefit you're going to see is if you are fetching multiple different files from the same domain. HTTP/1 would open and close a new connection for each request and had larger headers which were sent and received. HTTP/2 multiplexing allows a single TCP connection for all data from the same domain.

I used it on a site which by necessity had to download (stream) 4-50 large files on many pages. With HTTP/2 multiplexing, we saw huge performance improvements, but that is a special case.

It won't hurt you to use it, and I'm sure it is a good learning experience, but it isn't a necessity.

<comment><AUTHOR> Thanks for that!

I understand the basic issues, but as replacing the entire OS on my server would be needed to get the required underlying support, plus the pain of dealing with otherwise-unnecessary HTTPS, I'd really need some significant gain to bring that work forward, ie it would hurt me at least a little to try to do it!

Some of my pages do have tens of images, some have hundreds (eg whiteboxes of thumbnails); at the moment I have pushed a little of that heavy load out to other CDNs but maybe I could bring some of it back with HTTP/2?</comment></comment>


Dozens Charged with Scamming 15,000 Victims Through Fake IRS Calls - happy-go-lucky http://www.npr.org/sections/thetwo-way/2016/10/27/499585590/justice-department-charges-61-people-in-fake-irs-phone-call-conspiracy <comment><AUTHOR> So the story leaves out a lot of the interesting details.. but has been in thr headlines in India.

It was a massive, coordinated operation by the Mumbai police teams who discovered this by accident. Almost 200 policemen took part in the exercise, but most of them didnt know about the mission (to prevent leaks).

Almost an Infernal Affairs like operation - [http://m.rediff.com/news/special/how-the-rs-500-crore- scam-w...](http://m.rediff.com/news/special/how-the-rs-500-crore-scam-was- busted/20161014.htm)

<comment> happy-go-lucky A company of 700 people that are morally corrupt and seemingly unapologetic.

</comment></comment><comment><USER2> _Scam Phone Calls Continue; IRS Identifies Five Easy Ways to Spot Suspicious Calls_

[https://www.irs.gov/uac/newsroom/scam-phone-calls- continue-i...](https://www.irs.gov/uac/newsroom/scam-phone-calls-continue-irs- identifies-five-easy-ways-to-spot-suspicious-calls)

Things the IRS will never do:

1\. Call to demand immediate payment, nor will we call about taxes owed without first having mailed you a bill..

2\. Demand that you pay taxes without giving you the opportunity to question or appeal the amount they say you owe.

3\. Require you to use a specific payment method for your taxes, such as a prepaid debit card.

4\. Ask for credit or debit card numbers over the phone.

5\. Threaten to bring in local police or other law-enforcement groups to have you arrested for not paying.

<comment><USER3> However the IRS will soon be contracting private debt collection agencies [1] which may use some of those methods.

[1]:[https://www.irs.gov/uac/newsroom/new-private-debt- collection...](https://www.irs.gov/uac/newsroom/new-private-debt-collection- program-to-begin-next-spring-irs-to-contract-with-four-agencies-taxpayer- rights-protected)

</comment></comment><comment><USER4> Similar scams are also going on with utility companies and the level of activity has definitely increased this year. The spammers spoof Caller ID for a utility company such as PG&E, then tell the mark that their electricity and gas will be cut off unless they pay a past-due bill. They try to hit me at least once a week.

[https://www.pge.com/en/about/newsroom/newsdetails/index.page...](https://www.pge.com/en/about/newsroom/newsdetails/index.page?title=20160407_pge_warns_customers_of_increased_scams_during_tax_and_election_season)

<comment><USER5> > The spammers spoof Caller ID for a utility company such as PG&E, then tell > the mark that their electricity and gas will be cut off unless they pay a > past-due bill.

This is one of the myriad of times where being poor is expensive. I _know_ this scam wouldn't work on me today. Why? My utility bills are paid automatically through credit card charges made by the utility. I get e-mails every month or two when the charges are made. If I don't see an e-mail around the usual time, I can log in and investigate before the bill is past due. Even if the charge messes up for some reason, I have the money handy to pay both the utility charge and the late payment fee if it's my error.

Someone who doesn't have as much doesn't have those choices. I remember when I made far less than I do now and, yes, it's entirely plausible that I would get a call from the light company demanding payment or they'll have the person standing next to my meter go ahead with the disconnection. So I _would have_ fallen for that scam in the past because it was plausible. It sucks that scammers prey on people who are in these kinds of vulnerable positions.

</comment></comment><comment><USER6> I received several of these annoying calls. I knew it was a scam but I was amazed at their brazenness; how can they get away with claiming to be from the IRS and demand money? Good to hear they have been charged.

<comment><USER7> >how can they get away with claiming to be from the IRS and demand money?

I feel like these sorts of scams increasingly involve calling/emailing people in one country from a different country. This is presumably because enforcement within any single country is becoming more and more effective. By targeting people in other countries you create a situation where multiple police forces need to cooperate which presumably means you are able to operate for longer before arrests start happening.

</comment></comment><comment><USER8> I'm so glad. I was one of the people they tried to target. I tried to keep them on the line and extract as much info as I could and reported it to the IRS. I feel bad for the people they actually got money from and I hope they get it back.

<comment><USER9> I honestly haven't met anyone that WASN'T called with an IRS scam this year. It's amazing how they were able to call so many numbers and it lasted so long.

<comment><USER10> My primary number has been a google voice number with a weird area code for the last 4 years, and I have gotten maybe 4 robotcalls in that time.

</comment></comment></comment><comment><USER11> It's a pretty effective scam for a few reasons:

-You call THEM back, which immediately establishes trust (since you called them!) and makes it easier for you to fork over information/payment

-It's pretty scary to hear that the IRS is suing you, since almost everyone thinks they may have done something on a past tax filing that wasn't completely correct

<comment><USER12> It is effective because nobody ever knows if they own the IRS money or not. When you have a tax system like the USA it is impossible to know what you owe.

</comment></comment><comment><USER13> $300 million? Hot damn. I wonder if they would have gotten away with it if they'd stopped when they were merely wealthy instead of millionaires.

</comment><comment><USER14> I got several of these calls. Transcript of the voicemail. It's comical. Sad that some fell for it. I'm just glad the local cops didn't show up and take me into custody.

>>>>> Mission team of IRS we have just received a notification regarding your tax filings from the headquarters which will get expired in the next 24 working hours and once again expired after that you will be taken under custody by the local cops as there are for serious allegations pressed on your name at this moment we would request you to get back to us so that we can discuss about this case before taking any legal action against you

<comment><USER15> That sounds like deliberate word salad. Maybe the effect is to hit with you as many words as possible and effect some kind of change in brain state like whatever it it is that Derren Brown [0] does.

[0] [https://en.wikipedia.org/wiki/Derren_Brown](https://en.wikipedia.org/wiki/Derren_Brown)

</comment></comment><comment><USER16> Here's the source complaint & indictment

[https://www.justice.gov/usao- sdtx/file/905837/download](https://www.justice.gov/usao- sdtx/file/905837/download)

</comment><comment><USER17> Couldn't we spend less than $300M on our telco infrastructure to prevent shit like this? Black listing rented phone numbers, preventing caller ID spoofing, etc...

<comment><USER18> Yea, let's perpetrate a scam like this to raise money to prevent scams like this from occurring.

Or do you think the money is going to come from somewhere else? Why should I have to pay for the ever increasing coddling of the American public?

<comment><USER17> Troll

<comment><USER18> No, just someone with a different opinion than you.

</comment></comment></comment></comment><comment><USER19> Here's a video of said calls:

[https://www.youtube.com/watch?v=CuVFcizDwqw](https://www.youtube.com/watch?v=CuVFcizDwqw)

<comment><USER20> Another one, where they tried to scam a tax professional:

[https://youtu.be/poFAzDCGLrI?t=36s](https://youtu.be/poFAzDCGLrI?t=36s)

</comment></comment><comment><USER21> I am still waiting for them to prosecute the fake business loan scams. I get calls daily with the message that I have been approved for a 250K business loan.

</comment><comment><USER22> Lol. The IRS doesn't call you when they demand money, they just empty your bank account or come to your house and take all your shit.

<comment><USER23> No they don't do that either. At least not as their first move.

</comment></comment><comment><USER24> "The scammers have "done their research" using publicly available information, including sites like Facebook, he said. They identify vulnerabilities to exploit."

Stop posting personal info on Facebook, Instagram and Linkedin. Best way to keep in touch is Whatsapp and any other personal messenger app.

<comment><USER25> > Best way to keep in touch is Whatsapp…

…which is owned by Facebook, and shares account information with the Facebook Platform?

[http://www.theverge.com/2016/8/25/12638698/whatsapp-to- start...](http://www.theverge.com/2016/8/25/12638698/whatsapp-to-start- sharing-user-data-with-facebook)

<comment><USER26> Are you suggesting these people work for Whatsapp?

You could make the news with that kind of info.

<comment><USER25> Not at all. Just pointing out the inconsistency in recommending that people avoid social media sites like Facebook... and use a tool that's owned by Facebook, and which shares account information with it, as a replacement.

<comment><USER26> But it is a different situation. The implication is people posted on Facebook and it wasn't Facebook who ran the IRS scheme by reading the data on the backend, the data was shared by the users. WhatsApp doesn't work like Facebook's main product, it is more like SMS for example.

So the ownership of Facebook is irrelevant in this case.

Now perhaps it is possible to argue if someone hacked into Facebook to steal all data. But I don't think that's the case here.</comment></comment></comment></comment></comment>


Choose SQL - <AUTHOR> https://stateofprogress.blog/choose-sql-d017cfc08870#.ug46nrfxh <comment><USER2> I've been using Postgres for the past 20 years, and it's basically almost always been the right choice. Even when it's not, it's a good starting point to explore the problem until everyone's _really, really_ sure that something else might be best for that niche.

<comment><USER3> Short lived, high frequency/throughput data (most likely some kind of non- essential logging), that gets accumulated on an interval in a different database would be such a case. Something like Redis can be the right choice there.

Also PostgreSQL is much more than a SQL database, it rivals an application server with its flexibility through extensions and supported languages.

<comment><USER2> Yes; antirez originally created redis with a logging thing in mind if memory serves... Indeed:

[https://groups.google.com/forum/#!msg/redis- db/DHBqQ7x4sOU/e...](https://groups.google.com/forum/#!msg/redis- db/DHBqQ7x4sOU/eH_2nB2RNiMJ)

<comment><USER4> If memory serves? Redis serves everything directly from memory.

<comment><USER5> Yes, but it has Append Only Files (AOF) that are similar to postgres' WAL and allow it to be durable. It can also fork periodically and write a copy of the database to disk. (It's single threaded and linux will COW when forking so it's safe and efficient iirc.)

</comment></comment></comment></comment></comment><comment><USER6> Although he author doesn't take into account that not all web apps are created equal and the SQL/NoSQL argument isn't black and white... I do find myself slightly nodding in agreement with what he says.

NoSQL doesn't reduce development effort. What you gain from not having to worry about modifying schemas and enforcing referential integrity, you lose from having to add more code to your app to check that a DB document has a certain value. In essence you are moving responsibility for data integrity away from the DB and in to your app, something I think is quite dangerous.

NoSQL has its place, but I do feel that it is a bit more hyped up than it should be.

<comment><USER7> > In essence you are moving responsibility for data integrity away from the DB > and in to your app, something I think is quite dangerous.

Why is this a bad thing? The code in my app can be tested and is much more expressive than some contraints in the DB.

<comment><USER6> The problem there is what if you have two different apps connecting to the same DB. When the DB enforces referential integrity both apps can be certain the data in the DB is correct and they wont be able to mess with it. When the apps have to take on that role each app must implement (i.e. unnecessarily duplicate) the same integrity checking functionality. In a complex environment where you might have different apps using the same NoSQL data store that becomes very difficult to manage.

</comment></comment></comment><comment><USER8> Yes! As a data scientist, an important point of my job is evangelizing the data consumer's point of view. It's easy for data producers (e.g. the developers who write the logging code that generates the data I use) to make decisions that make their lives easier, at the expense of making my life much harder.

Only by talking to each other can we find the balance that mutually optimizes our joint effort level. A little more effort up-front to log data in a format with well-defined schemas, translates into huge savings in processing, consuming, and understanding the data down the road.

<comment><USER9> _> evangelizing the data consumer's point of view_

Note that this is a fundamental principle in software development, not just for data science:

        Prefer data structures that are simple to interpret over those simple to generate.

This principle follows from two basic observations:

1\. Data is usually produced by one application, but consumed by many different applications.

2\. Writing a serializer is almost always easier than writing a parser. [1]

However, it is not always clear how to apply this principle. For example, as a library or web service author, you define the API on your own, rather than having your users define it. It is extra effort to get feedback from your library users, and to design your API accordingly. That's why most APIs are geared towards the data producers rather than consumers.

There is even an enterprise design pattern for this:

[https://martinfowler.com/articles/consumerDrivenContracts.ht...](https://martinfowler.com/articles/consumerDrivenContracts.html)

[1] The former usually involves a template system and a few helper functions. The latter usually involves regexes embedded in ugly code, parser generators with all their shortcomings (and the fact that most programmers don't bother to learn using them), complex DOM or JSON traversals that fail to handle all special cases, or masses of autogenerated classes automatically derived from some XML schema (or JSON schema).

</comment></comment><comment><USER10> > Second, no; NoSQL does not scale better than SQL. At least not in the manner > you think. MySQL is being used from top-traffic websites like Facebook and > Twitter, to high traffic websites like GitHub and Basecamp and almost every > PHP installation out there. If this is not scaling, then what is?

YouTube, mentioned in the article, also uses MySQL, but the author forgets to mention those big names don't use a plain simple Vanilla SQL. In the case of YouTube, it's Vitess [1]. It would be nice for the author to mention this.

[1] [http://vitess.io/](http://vitess.io/)

<comment><USER11> All those big sites do a lot of things to make MySQL scale.

The point is just fighting the perception that MySQL can't scale.

<comment><USER12> > The point is just fighting the perception that MySQL can't scale.

Which is a misplaced point that'd misleading an entire generation of engineers.

MySQL doesn't scale. The companies mentioned had to throw huge amount of engineering resources to make it work, whatever the costs and the drawbacks. They had no choice left, hack it or die out of your own traffic.

<comment><USER13> How many systems are out there that will run at YouTube/Facebook/etc scale without "huge amount of engineering resources to make it work"? Zero. Zilch. Nada.

<comment><USER12> System that will run at hundreds of times the scale of MySQL without requiring the equivalent hassle in engineering:

Cassandra, ElasticSearch, Riak, RedShift, BigQuery, DynamoDB, BigTable, Spanner, Terradata, S3...

<comment><USER14> systems that are designed as distributed datastores run as managed, hosted services on MASSIVE cloud datacenters...yea those will scale.

if you're trying to keep your hosting overhead down though none of those is available (except S3 I suppose, which is super cheap, but also not really a datastore solution) but a basic app server running MySQL (or similar) is. now, I'm not saying anybody should expect scaling for free. I'm just saying you're comparing apples to oranges here.

The apples to apples comparison is basic app server running MySQL vs. basic app server running MongoDB. the myth is that the MongoDB instance inherently scales better than the MySQL instance. it doesn't, particularly. it has better write performance, but not much else that gives it better scalability.

and the real point of the article is to combat some of the magical thinking that has lead to many bad decisions from misinformed developers who heard a rumor that a NoSQL datastore has orders of magnitude better performance characteristics than an RDBMS.

<comment><USER12> When we talk about scaling (not necessarily to the size of reddit/youtube) that implies you're already way beyond an app running with a mysql.

> MongoDB

MongoDB is a pile of crap and a shame to all the NoSQL datastores. Please don't bring MongoDB in a talk about databases. If that is your only experience of NoSQL, it is perfectly normal that you think of NoSQL as a mistake to never use ;)

<comment><USER14> No need to convince me of Mongo's problems. I struggle to think of a use case for it where I wouldn't prefer to use the JSON store functionality of Postgres instead.

My best experience with NoSQL (in terms of it working ok and not otherwise fucking up the system) has been with HBase on a cluster running map/reduce jobs. That was well suited for the purpose (mostly). I'm sure glad nobody had any illusions about using that as a production datastore for a user facing web app though.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER15> SQL databases are very powerful and capable tools, but they also hide some complexities behind a declarative language. That makes easy to get into a bad place, performance-wise, if the DB mutates quickly. You don't need millions of accesses to the DB to have unbearable slow queries if schemas are not carefully designed, which is difficult in early stages of development.

I usually consider the "I had to migrate from DB X to DB Y" a success story. You discovered what was required and later in the process, improved your system. Painful, but probably unavoidable.

Databases are not easy to handle and operate. Is easy to underestimate them. We used to have people just devoted to them (can you remember DBAs?). It is worth to spend time learning them and knowing their goods and bads. And sometimes making mistakes in unavoidable...

<comment><USER16> I'm a dba so yes I can remember DBAs. I don't understand the condescension from you comment. As a DBA, I have never understood the argument of a mutating database as a solid justification. It just sounds lazy/unprepared, at best. Sure, there may be a valid complaint for the unicorns out there, but they represent such a small percentage of situations that I hardly see how ther experiences necessarily translate to everyone else's situation. "It's hard" or "it's difficult" sound like blame-shifting to cover for a lack of actually understanding requirements.

I don't mean to single you out in this comment. The reality is that your comment just really reiterates so many of the same arguments that are regularly trotted out as justification.

NoSQL already came and went. It was called ISAM. It has a use case, but it's not the silver bullet everyone wants it to be.

Oh yeah, as a DBA, I've loved hearing how NoSQL was going to do away with my job and that I am a 39 year old dinosaur who'd better catch up and get on board with DevOps. Guess who runs data CI where I am? Guess who manages our Mongo and Redis servers. Yep. The ancient DBAs.

<comment><USER17> How about some vitriol?

You know what the #1 feature of nosql is? NoDBA.

I can hear you screaming you'll still need a DBA. You don't get it. You won't need an Enterprise DBA.

Enterprise DBAs are almost invariably horrible blockades to data storage in any enterprise. All they do is lock away the database from developers, cram Oracle down their throats, do things as slowly as possible behind ticket walls. You lock the data away pining for your halcyon mainframe days of white coats and clean rooms, and force your slow ponderous overtenanted infrastructure.

Your profession, since you have proudly identified as a DBA, you are an Enterprise DBA, is a disgrace.

<comment><USER18> To add to this point as well; agile methodologies work best when everyone is a generalist and the business is onboard with being involved in the development process.

DBAs are specialists which makes them inherently siloed.

<comment><USER16> I find your perception of DBA work to be quite narrow. Too many comments on here are based purely on colloquial evidence and not enough actual communication and understanding.

</comment></comment></comment></comment></comment><comment><USER19> Choose the right tool for the right job. We are all building unique systems solving unique problems. SQL is good for some things and NoSQL is good for some things.

If you are taking the time and (someone else's) money to build something do it right and research how various data stores work up front. You should be at the point where it is obvious what the right decision is in your mind and if it isn't...get your learn on until it is.

<comment><USER20> And the thesis of this article is that SQL is overwhelmingly the more versatile and better choice.

<comment><USER19> Choice for what? The article talks about a very specific domain (web applications solving a certain class of problem). Many organizations use multiple data stores and handle different problems.

At Goldman Sachs there is almost every imaginable type of data store in use. All carefully vetted decisions that are largely the right choice much of the time. Almost all of it flows into some form of web application at some point in time in a giant interconnected system.

I am very happy we don't use a single class of data store. If we did we'd be fucked.

The "thesis" may apply very narrowly to a small web app of a certain size. But honestly if you are building something that cookie cutter are you really building something unique? What you are doing has probably been commoditized.

</comment></comment></comment><comment><USER21> We had key-value stores before there were relational databases. RDBMSs are built upon them, in fact. Now a new generation discovers the reason that RDBMSs became popular: they offer a more general purpose abstraction that is more useful outside of certain specific cases.

</comment><comment> joatmon-snoo > MySQL is being used from top-traffic websites like Facebook and Twitter, to > high traffic websites like GitHub and Basecamp and almost every PHP > installation out there.

Is it just me, or does this seem rather misleading? My understanding is that they can use MySQL because they've carefully designed their architectures to direct most hits to caches or Memcache/Redis, which minimizes the load on the database.

<comment><USER4> How is that misleading? Using RDBMS as your primary data store is the decision that people reading this are going to benefit from.

When it comes time to cache and optimize for things like search, that's when you use the additional technologies.

It's a huge mistake to think that companies that are using Redis/ElasticSearch/MapReduce are doing so to cover up for the wrong decision they made of going with an RDBMS. The RDBMS is their source of truth, everything else is just an indexing mechanism.

<comment><USER12> The RDBMS doesn't have to be the source of truth. Other databases can do that just fine.

</comment></comment></comment><comment><USER22> I'm currently at my second startup doing a migration from Mongo to Postgres.

<comment><USER23> [https://github.com/stripe/mosql](https://github.com/stripe/mosql) was very useful in my conversion at work. (That said, we only had one small service on Mongo at the time, but it trivialized the conversion effort entirely nonetheless.)

</comment></comment><comment><USER24> I've been thinking about this a lot recently, and while I used to unambiguously agree with this, I'm slightly more uncertain now.

First, to be sure, in the year of 2017, it's definitely the safe path to go with Postgres or MySQL. Traditional relational databases aren't distributed out of the gate, which is both a good thing and a bad thing. They're mature, which means that lots of bugs have been found and fixed, and there's a Stack Overflow answer for every basically error possible. And yes, Postgres will remain the "best" choice probably for at least another decade.

Very broadly, traditional relational databases are chosen because of two advantages: features (rich data model, query language, ACID transactions, etc.), and maturity. NoSQL datastore generally sacrifice the former for the sake of scalability, and lack the latter simply because they're too new.

First, the features. I think most people are in agreement that stripping down your data model to fit into a basic but highly scalable key-value store ultimately hurts productivity. But it's definitely not impossible for a scalable distributed database to provide most or all of the features that a traditional relational database provides; that's exactly what Google did[1], and what some open source projects[2] are trying to emulate.

Sharding Postgres is safe, but it's also hard. It does messy things to your application code, and _also_ hurts productivity. And at the end of the day, you _still_ don't get ACID transactions and joins if you ever have to do anything that touches more than one shard. We really do need a better solution that combines the best of both worlds.

Sure, traditional relational databases will probably always be the most mature databases around. But there will come a time when those new databases are mature _enough_. And I think along the way, we can rethink some of the properties of traditional SQL that make it hard to work with sometimes.

1: [https://static.googleusercontent.com/media/research.google.c...](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41344.pdf)

2: [http://db.cs.cmu.edu/papers/2016/pavlo-newsql- sigmodrec2016....](http://db.cs.cmu.edu/papers/2016/pavlo-newsql- sigmodrec2016.pdf)

</comment><comment><USER25> Agree mostly with the article; nitpicks:

>In the first case there is a language using words that humans use to talk to each other, while in the second one there is pure JavaScript, where you have to build your query using a JSON object and even convert the 24 hours to milliseconds on your own.

I have never personally implemented anything using either MySQL or Mongo. But even I can see that this is a bad example. In the first case, you have an SQL query _raw_. This obviously has to be passed to some sort of API to be used in code, so it's an incomplete example. In the second case, you're blaming what appears to be a weakness in Javascript's "Date" library on MongoDB. In Go this becomes time.Now().AddDate(0,0,1) which is relatively straightforward. nb. Go time lib becomes less pretty whenever there's an error involved and I sort of hate it honestly.

>First, scaling is not and most probably won't be your problem. Unless you start having a few thousand of queries per second and terrabytes of data in your database, this won't be a problem at all.

...true...

>And in case it does, you can go with a fully managed solution like AWS' RDS.

...but the freshman delusion is "I have scaling problems", so the sophomore delusion is "I will never have scaling problems" :p

The job of a businessperson includes managing things. You can't outsource all of the management. It's how you add value after all. I've seen plenty of cases where people choose not to go with AWS because they can save money by doing more work and they made off with some dough.

<comment><USER26> > you're blaming what appears to be a weakness in Javascript's "Date" library > on MongoDB

But this is the point: that SQL is a better query language than Javascript (which is what MongoDB uses).

</comment></comment><comment><USER27> The biggest problem with Mongodb is that it _loses freaking data_ all the damn time. I never fully realized why it's so hated before taking my current job, which uses it as the primary database, distributed among a bunch of servers. There are all sorts of concurrency issues all the time. And no, write- concern:majority doesn't help.

</comment><comment><USER28> I use both. I even use text indexing, redis, queues and other things.

Sometimes RDB is better. For other things it's easier and more scalable with document store.

They need to know about each other using your application logic (join by GUID for instance), but you're doing that anyway even if you religiously staying in one camp or the other.

</comment><comment><USER29> Warning, rant incoming: I still want to do uncivilized things with the <insert swear word here> who took away WebSQL and left us with the <insert more swear words> NoSQL <...> that is IndexedDB.

No JOINs, no natural way to express a ORDER BY x LIMIT y,z... no way to e.g. mirror your back-end database in order to implement stuff like full-featured offline/online apps... and don't get me started on the fact that IndexedDB is async which makes even emulation of the features I stated an open invitation into callback hell. Or, basically, ANYTHING more complex than a SELECT x FROM y.

How could this ever end up being a freakin' _standard_ and WebSQL being dropped?!

(If anyone here decides to implement a WebSQL equivalent backed by IndexedDB, here are my monies, take them! Please!)

<comment><USER30> I'm not sure WebSQL was but maybe it just died off as nobody was using it?

<comment><USER29> WebSQL was basically SQLite accessible from JS - which also, unfortunately, was the reason for it being dropped... apparently the standards committees only want standards with multiple competing implementations.

Which is total bananas, given the fact that there's only one (free, ultra portable, tiny) embeddable SQL database - SQLite. Which every browser except IE/Edge ships, anyways, for internal data storage.

<comment><USER31> I'm a big fan of sqlite and would have loved to see WebSQL succeed, but I think HTML adopting the de-facto behaviour of sqlite is "bananas" \- it would be the equivalent of "formatLikeWord95" of the OOXML spec. They should have defined a proper ISO SQL subset/profile instead.

What remains of WebSQL is the commonly implemented SQL client driver API for Node.js which isn't so good a fit IMHO.

<comment><USER29> > They should have defined a proper ISO SQL subset/profile instead.

That wouldn't have helped. Google Chrome (and, by extension, Safari and Opera) and Firefox would just use the already present SQLite library in the engine... and I think MS would also have chosen SQLite instead of risking to develop a full SQL database (and opening a rich source of bugs and security issues, in contrast with using battle-tested SQLite). Again, just one implementation of the standard.

<comment><USER31> Or, MS would have blessed us with their Access/JET DB.

<comment><USER29> Well if MS could build a safe interface to Access (also managing translation from ISO SQL to Access SQL!), I'd be perfectly fine with it! I've seen people building really nice stuff based on it, a pity it's only available on "higher end" Office variants and even there only on Windows :(

</comment></comment></comment></comment></comment></comment></comment><comment><USER32> Tools that we use influence our decision making, shape our experience and ultimately architectures of our systems. Different constraints lead to very different designs.

And using databases designed in the nineties just cannot get you to fast resilient services or to any relevant experience necessary to build them at some point in the future. But it will waste just as much time if not more. CRDTs and eventual consistency are actually easier to understand and reason about, than transactions. Don't get fooled by familiarity. Don't choose MySQL, choose Cassandra, Riak, DynamoDB, don't choose POSIX storage for the same reasons, choose object storages, Swift, S3, etc.

<comment><USER13> > CRDTs and eventual consistency are actually easier to understand and reason > about, than transactions.

I suspect we're going to have to agree to disagree on this one. Without spending too much time on this, it's pretty easy to cite the Google F1 paper ([http://static.googleusercontent.com/media/research.google.co...](http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/41344.pdf)):

"The system must provide ACID transactions, and must always present applications with consistent and correct data. Designing applications to cope with concurrency anomalies in their data is very error-prone, time-consuming, and ultimately not worth the performance gains."

</comment></comment><comment><USER33> Since the title is a bit misleading (talks about database choice: SQL vs NoSQL), I comment about the big data/data analytics point of view in case anyone comes to this post for the same reason.

The only problem I have with SQL (I use Postgres, in the form of Redshift almost daily: it's excellent when it works as expected) is the not-great readability of the code and the abysmal testing/debugging/checking of SQL code. Have had to write lots of boilerplate (Python, mostly) for that, and it feels like we are reinventing a wheel someone else is using, somewhere.

<comment><USER34> > is the not-great readability of the code

Isn't that more of a poor-SQL-programmer problem, than a SQL-Database- technology problem?

> and the abysmal testing/debugging/checking of SQL code

I haven't used Postgres, but MS Sql Server has a ton of testing, debugging tools including SQL Trace, Explain Plans, and even a Stored Procedure debugger wherein you can set breakpoints and step through your code.

Of course, one cannot expect the same level of rich debugging features that one gets with traditional IDE for programming languages, since SQL programming is a somewhat different beast in that sense.

<comment><USER35> SQL Server's tools are awesome! Too bad the licensing fees are not.

<comment><USER36> All that tools are included for free.

You are only limited if you try to use a higher-end feature (like analytics) but the overall tooling is all there.

</comment></comment></comment></comment><comment><USER37> Yes so much yes. Be sure you need NoSQL when you replace SQL. SQL is so powerful, mature and reliable. If your problem can be solved with plain old SQL, it's probably the best choice.

<comment><USER38> NoSQL developers can command higher pay because there are fewer of them. Same with any other shiny new tech, really.

<comment><USER37> Yeah I've also heard good COBOL developers can command huge pay on legacy banking systems. That would be a good reason to learn COBOL, but I don't think it makes COBOL a good technology choice.

There are times where NoSQL is a better choice. When massive scale is involved I understand that NoSQL can come into it's own. I'm sure there are other cases as well. But when it gets used for CRUD, it tends to be a bad call.

<comment><USER12> COBOL is good at what it was intended for. Just like NoSQL.

</comment></comment></comment></comment><comment><USER39> Glad to see this post.

One other thing I'd add - I read a very good mantra for software development, that I've adapted a bit in my head.

Your data will outlast your application

Your application will outlast your framework

Your framework will outlast your developers.

For many of the apps I write, the database is the core of the application. One question I like to ask myself is: if my users were well versed in UNIX, SQL, and a programming language, how well could they get along without the web application?

The answer to this question reveals how well I've designed my app.

For instance, is the backend database useful on its own? Or is it merely a persistence tier for scattered bits of information that only become useful once assembled by code? If it's the latter, then no wonder people start questioning the value of SQL. It isn't functioning as it should, there's no relational structure to the data. Persistence for scattered bits of objects that need to be reassembled through code means that the relational database is almost just getting used to serialize objects back and forth to disk. If you're doing that, then of course you start to see SQL as just a bunch of stuff that complicates things! I mean, at that point, why not just write and retrieve objects? There's almost no difference. Thing is, this sometimes reveals a mismatch between SQL and the problem at hand, but just as often, it reveals a developer who doesn't really think in terms of databases and relations, and who just sort of crams things into a SQL database because it is the default persistence tier.

Is the code that does essential analysis and logic on the data easy to run and understand on its own? If my users were capable of logging onto the machine, running this code as command line scripts, and reading the output, how useful would it be to them? How clear and concise is the code?

If both tiers are clear and easy to understand for someone grounded in SQL, Unix, and a programming language, I have control over my code base. If not, there better be a really good reason. Eventually, the developer will leave, the framework will be out of date, and someone new will need to work on the code base. If you can't make sense of the database, in particular, you are in serious trouble.

<comment><USER40> Please someone have this reply pinned somewhere always visible. Great mantra/wisdom.

</comment></comment><comment><USER41> This is just another article telling me about what I should do with my web application without asking me anything whatsoever about my web application. Just because you made a bad choice for your situation doesn't necessarily mean everyone else did too.

<comment><USER42> This is just another comment assuming that the article is talking directly to me. Just because it makes some generalisations that are useful to many, doesn't necessarily mean it is tailored to my exact situation.

<comment><USER41> Come on... The article is clearly written for a general web dev audience (if not, please let me know the more specific audience you've inferred) and starts with this completely unqualified first sentence:

> Let's get straight to the point; choose an SQL database for your web > application. I think I can't make my self clearer.

Yeah there is better qualification later, but there are whole paragraphs of railing against nosql databases for all kinds purposes that they actually CAN make sense in, for certain applications.

</comment></comment></comment><comment><USER43> With things like Citus and Memsql readily available, it really makes the scaling part of relational databases so much easier.

</comment><comment><USER44> What are your thoughts on InfluxDB or Cassandra to model and use for apps? They are two "NoSQL" solutions that seem to strike a balance of simplicity and query capability. The only NoSQL solution I'm experienced in is RIAK which is purely key/value and I find SOLR (which aggregates for RIAK) not that friendly.

</comment><comment><USER45> There are alternative syntaxes to SQL that I find easier to maintain. Like sequence of piped commands like bash or Splunk/Sumologic/Azure analytics language and similar.

Order in syntax for SELECT is weird. I usually look FROM then WHERE then the fields being selected. Feels like reading backwards.

</comment><comment><USER46> Although I agree with most of the author's points I think he underestimates the value of ease of use by programmers. SQL syntax can definitely be daunting to some and difficult to understand complex queries. There is room for improvement in that regard.

<comment><USER47> Take your favorite complex query. Write code to accomplish the same thing with raw data in whatever format you consider convenient.

You will be AMAZED at how compact SQL actually is by comparison!

<comment><USER48> Indeed.

An often overlooked point is that good SQL optimizers generate different versions of queries depending on the current state of the database and the actual parameters to the query. That is, the optimal query plan for finding the sales to "school teachers" is probably different than for "astronauts" and the optimal query plan for reporting on "todays orders" is probably different at the start of the day than late in the afternoon. SQL query planners try to do the right thing with this sort of data variability and generally (but not always!) do a good job. It would take an immense amount of work and insight into the data and all the potential use cases to approach this with custom coded queries.

Before there were relational databases these sorts of access path, query strategy decisions had to be made by developers and were reflected in the schema and physical design of the database. This was inflexible and very labor intensive and it made many kinds of applications and or even routine changes uneconomical.

<comment><USER47> Oh trust me, I don't overlook that point. I've spent too much time cajoling optimizers into the query plan that I know will work when they think they know better. And I know from painful experience how much more work it is to cajole developers in the same way.

That said, there is little point discussing this subject people who clearly have no experience about how to do what a little bit of SQL does. If you're convinced that manipulating data is easy, you're not going to appreciate that manipulating data efficiently is even more tricky.

</comment></comment></comment></comment><comment><USER49> It's not a matter of SQL/something else, but what you're storing in your DB, how you model data.

If you're storing state rather than information (append-only), you'll have a hard time in the future regardless of query language.

</comment><comment><USER50> Agreed, but for _god 's sake_ don't choose MySQL.

</comment><comment><USER23> I think SQL is a great basis point for awesome databases, but I don't think it's necessarily the only option we should have. Options are good!

<comment><USER51> It's good to have options; it's dangerous to believe all options are equivalent. For 99.9999% of applications, you'll have a much better time starting with a SQL database. If you've somehow end up with an application in the 0.0001% you can probably safely convert to something else if needed.

</comment></comment><comment><USER52> Isn't comparing NoSQL and SQL servers a little simplistic? Surely there is more than one type of NoSQL server...

<comment><USER20> And more than one kind of SQL server.

Redshift and SQLite are pretty different.

</comment></comment><comment><USER53> [deleted]

<comment><USER23> NoSql solves a different problem set. Modern SQL servers definitely have a lot limitations at huge scale. You can only write-scale so far before needing to resort to the variety of fragile SQL scaling solutions. (Read scaling has more options).

That's not to say they lose usefulness at scale or should be avoided because they'll be a "bottleneck" at some point (if a SQL database is ever a bottleneck, you're super lucky!) but there are definitely cases where a non- SQL database is the right choice.

In some cases, it's not a scaling but just a use-case difference (Elasticsearch)

</comment></comment><comment><USER54> Hipster Driven Development is really expensive.

<comment><USER55> Not much when combined with StackOverflow-Based Programming.</comment></comment>


Reverse-Engineering WatchGuard Mobile VPN - <AUTHOR> https://www.tazj.in/en/1486830338 <comment><USER2> Would that actually be a vuln? I've got several clients using it for VPN.</comment>


NSA spying hurts business of large U.S. hardware makers - <AUTHOR> http://www.usatoday.com/story/tech/columnist/shinal/2013/12/08/nsa-spying-has-damaged-overseas-business-of-strongest-us-tech-companies/3891765/

<comment><USER2> The crisis of trust in US telecom gear hits the mainstream media. Cisco should be way ahead of a columnist in USA Today in both diagnosing and addressing the problem, right? Read the conference call transcript. Only one mention of "NSA" and that is in an analyst's question that was deflected.</comment>


Happy Birthday, Apache Group and "The Apache Way" - <AUTHOR> https://blogs.apache.org/foundation/entry/the-apache-software-foundation-celebrates1 <comment><AUTHOR> $20B+ worth of software for the public good at 100% no cost

350 Open Source projects

730 individual Members, 7000 Committers stewarding 200M+ loc

> six dozen Apache projects form the foundation of $166B Big Data ecosystem

Happy Bday @TheASF!</comment>