Pixar's Rules of Storytelling - <AUTHOR> http://aerogrammestudio.com/2013/03/07/pixars-22-rules-of-storytelling/ <comment><USER2> _Once upon a time there was ___. Every day, ___. One day ___. Because of that, ___. Because of that, ___. Until finally ___._

Also a good way to write a product definition.

<comment><USER3> This is great! Although I'm confused about the two Because of Thats... can someone provide an example of how to complete this in a way that this part makes sense?

<comment><USER4> Act II of your story is a chain of not just one cause and effect events.

</comment></comment></comment><comment><USER5> Emma Coates originally shared this on her blog: [http://storyshots.tumblr.com/post/25032057278/22-storybasics...](http://storyshots.tumblr.com/post/25032057278/22-storybasics- ive-picked-up-in-my-time-at-pixar)

<comment><USER6> Thank you _grumble grumble blogs that don't link back to original content_

<comment><USER7> To be fair, I don't think they realized that it was on her blog. Both her blog entry, and the linked article were aggregated from her Twitter.

</comment></comment></comment><comment><USER8> The article had a link to a "related post" which had Neil Gaiman's 8 rules for writing: [http://aerogrammestudio.com/2013/02/28/neil-gaimns-eight- rul...](http://aerogrammestudio.com/2013/02/28/neil-gaimns-eight-rules-of- writing/)

I found those far more useful than the Pixar ones. (Although I admit that "when stuck, list everything that would NOT happen next" is a pretty good one.)

<comment><USER9> "Perfection is like chasing the horizon." \- Neil Gaiman

Great quote, applicable to nearly everything.

</comment></comment><comment><USER10> Previous discussion: <http://news.ycombinator.com/item?id=4086805>

</comment><comment><USER11> As suggestions to tape on your wall, these seem excellent, something to look at when you are stuck. As rules, maybe not.

"What is your character good at, comfortable with? Throw the polar opposite at them. Challenge them. How do they deal?"

Interesting notion, but then I think of _War and Peace_. Pierre Bezuhov is good at nothing much at all. He is challenged, and transformed. But what is the polar opposite of being a prisoner of the French in Moscow and during the retreat? Prince Andrei is good at a hell of a lot of things; what seems to transform him is being shot up to one degree or another...

<comment><USER12> First you think you're above the rules, then you know enough to follow them, then you know enough to break them.

</comment></comment><comment><USER13> "Come up with your ending before you figure out your middle. Seriously. Endings are hard, get yours working up front." Mass Effect 3... 17 suck endings. That got to be a record somewhere.

<comment><USER14> Even further on this tangent -- Mass Effect's problems began right at the setup. It starts too big. You begin as a hero, chosen by Earth to be, in essence, the representative for all humanity. (Recall your choice of back stories each points to your character having a past more interesting than anything that happens in the actual story -- always a bad sign and indicative of lazy writing.) It's a terrible way to start a story, it's a cheap trick (raising the stakes by giving them really big important labels). It's a testament to the game's execution that it is as good as it is despite this quite awful setup.

<comment><USER15> Given the scale of the problem (reapers), starting at a Spectre scale is pretty logical. Else you'd have character progression that is too fast

<comment><USER14> They picked the scale of the problem too.

</comment></comment></comment></comment><comment><USER16> This is great. It also makes me think that, for whatever reasons, Cars 2 was not written by Pixar.

It's a horrible sequel. The original story had fully rounded characters, personal discovery, redemption, recognition of working with others, a whole pile of admirable qualities.

Cars 2 has a bunch of animated effects, celebrity cameos, and lots of guns, shooting and rockets. There may have been a story with a message but I'm not sure what it was.

Would it have been that hard to come up with a better story for the follow up?

<comment><USER17> Cars was quite profitable post-box office, I can't help but think Disney pushed for a sequel before the buzz ran off.

<comment><USER16> To my cynical eye, Cars 2 was written solely from a merchandise expansion point of view.

I just don't understand why someone didn't stand up and say 'can't we write a good sequel and make lots of money?'

In short : why not Empire Strikes Back? Lots of cool merch in that one, but a good story as well. It can be done.

</comment></comment></comment><comment><USER18> Bujold's method of plotting is to start with characters that her readers care for, and then ask: What's the worst thing that can happen to them?

And then do that.

</comment><comment><USER19> I like the old Disney cartoons better, and they obviously weren't written using those rules. I feel that nowadays movie get more and more similar. Could we still get something as atypical as the Robin Hood cartoon or Snow White by following those rules?

</comment><comment><USER20> It's not bad but it feels too formal and restrictive. 23rd rule is that there are no rules for an author to create something really great.

</comment><comment><USER21> It is interesting how replacing character with object makes it a decent guide to figuring out problems or refactoring in programming.</comment>


Andrew Mason's $378.36 Retirement Package - <AUTHOR> http://finance.yahoo.com/news/andrew-masons-378-36-retirement-190743504.html <comment><USER2> "Mason sold around $28 million worth of shares before Groupon's IPO in the fall of 2011."

I think he'll be ok.

<comment><USER3> "Much of Mason's wealth comes from Groupon's stock. He owns 7 percent, or about 46 million shares, according to FactSet. Based on Thursday's closing price, that's worth more than $208 million."

I think he'll be more than ok.

[http://www.npr.org/templates/story/story.php?storyId=1731820...](http://www.npr.org/templates/story/story.php?storyId=173182095)

<comment><USER4> as GRPN's share price went up almost 20% since his announcement - well I guess its fair to say he made about $40M (on paper) by leaving.

<comment> grey-area As long as he sells soon and realises his gains, otherwise he might end up with nothing. The money isn't real till the shares are sold.

</comment></comment></comment></comment><comment><USER5> > He can take comfort in knowing that, by leaving, he has already boosted his > net worth.

Thats so sad!!!!

<comment><USER6> And yet, at the end of the day, he has $200-300 million. I don't feel too bad for the guy.

<comment><USER5> I dont know about you, but a lot of the joy that I get out of building things has nothing to do with money.

</comment></comment></comment><comment><USER7> Does anyone know where he came up with the magic number? "Mason's annual salary is $756.72"

<comment><USER8> Six months is half that - i.e. the number in the title.

The salary is disclosed in public filings.

<comment><USER9> I think the question is about the genesis of the unusual annual salary of $756.72.

<comment><USER7> Yes, that was the question. Thought it might be a multiple of some arcane physical constant.

<comment><USER8> The answer will probably pop up on Quora soon - behind a registration wall no doubt.

<http://www.quora.com/Groupon>

And yeah, it could be a number broken down per year, per month, per week, per day, per hour.

</comment></comment></comment></comment></comment><comment><USER10> I call "linkbait"</comment>


Internet TV moves from the computer to the living room - <AUTHOR> http://www.economist.com/science/displayStory.cfm?story_id=13562114

<comment><USER2> _At present, the slickest is unquestionably the $149 box from Vudu_

The Vudu box is actually a rather interesting case. From the start, their marketing team wanted to be able to offer relatively high-quality 1080p movies, with audio, at 3 megabits per second--enough to play in realtime on an ordinary broadband connection without prebuffering.

In almost any other situation, this would have been a total disaster: 3 megabits for 1080p is practically impossible with current technology. However, a few things saved them:

1\. They realized that you don't actually need to use constant bitrate for streaming. If you have a local hard disk for buffering, you can let the bitrate variance increase over time by slowly buffering ahead of the main movie, so that once you're a few minutes in, you have a very large buffer within which to allow variable bitrate.

This is the extra benefit of having local storage: a normal VOD service forces you to use constant bitrate for the entire stream.

2\. They used H.264 video encoded with x264.

This is the best case I've ever seen of the fortunes of a company practically resting on a single open source program. They tried all sorts of solutions, and x264 was literally the only option that gave reasonably viewable output at the rates their marketers insisted on.

(Disclaimer: I have worked for Vudu and I develop x264 ;) )

IMO, this is going to be one of the make-it-or-break-it factors in the video on demand business: can you provide good enough video quality to satisfy people? Nobody cares much about quality when watching Youtube, but in order to convince someone to _pay_ for your content, the bar is vastly higher; you can no longer get away with your streams looking awful.

Many VOD services have failed because of this: if someone is going to pay the price of a DVD rental to watch a movie, they want it to look as good as a DVD. This is exacerbated by the fact that the initial adopters of such services are going to be more technical users who are willing to spend significant amounts of money on their home entertainment system: the same people who will be turned off by Youtube-like video quality.

This is also the challenge of competing with television services: while companies like Comcast prove again and again that having tons of bandwidth doesn't mean you can't completely cock up your video quality, it's vastly easier to do a good job when you have a gigantic pipe than when you're running over someone's low-end DSL.

</comment><comment><USER3> "Your correspondent thinks Boxee will be one of the most disruptive things to happen to television in ages—and not before time."

Can someone explain this final sentence to me? What does, "-and not before time" mean?</comment>


Microsoft CEO Steve Ballmer's Full Memo to the Troops About New Reorg - <AUTHOR> http://kara.allthingsd.com/20080723/microsoft-ceo-steve-ballmers-full-memo-to-the-troops-about-new-reorg/ <comment><USER2> The arrogance/cockiness/delusion levels throughout this memo are off the charts. Worth reading for the description of google as a "white page with 10 blue links" quote alone.</comment>


Why 'I Have Nothing to Hide' Is the Wrong Way to Think About Surveillance - <AUTHOR> https://www.wired.com/2013/06/why-i-have-nothing-to-hide-is-the-wrong-way-to-think-about-surveillance/ <comment><USER2> I am surprised that this Wired article didn't mention anything related to Snowden, considering he was the one who blew the lid on mass surveillance. Edward Snowden summed this up perfectly:

>Arguing that you don't care about the right to privacy because you have nothing to hide is no different than saying you don't care about free speech because you have nothing to say.

<comment><USER3> > ...article didn't mention anything related to Snowden, considering he was > the one who blew the lid on mass surveillance.

I really wish people would look beyond Snowden. It isn't as if there was zero knowledge of these types of programs or whistleblowers before him. William Binney, Thomas Drake, J. Kirk Wiebe, and Edward Loomis, just to name a few, deserve a lot more credit in this instance.

<comment><USER4> Published evidence goes back to the eighties, but made virtually no difference - until Snowden blew the lid off. Respect to all the others, yup, but Snowden broke the issue wide open.

<comment><USER3> You need to reevaluate your definition of that idiom.

<comment><USER4> You'll have to disambiguate (two idioms above) and specify your advice. References welcome.

<comment><USER3> The idiom "blow the lid off" means to publicly expose something which was previously secret or hidden. As you said yourself, evidence for these types of programs has existed for decades. Therefore, Snowden did not blow the lid off anything. What he did, instead, was to present corroborating evidence which added much needed detail in regard to such programs. I'm not trying to devalue his effort, just pointing out that he isn't some lone ranger. And the fact that so many people payed attention this time around had a lot more to do with the criminality of his actions rather than the information he presented.

<comment><USER4> Alas it this knowledge was hidden from the vast majority of people who rely on TV news for a long time. "Blow the lid off" doesn't mean the first person to reveal something, it refers to an explosion that can't be ignored. Kudos to the rest, but he was the one who really rang the gong.

Many revealed the Holocaust to the Allies during WWII, but very sadly, no-one managed "to blow the lid off" until Germany collapsed. "The Nazis attempted to keep the Holocaust a secret, but in August 1942, Dr. Gerhart Riegner, the representative of the World Jewish Congress in Geneva, Switzerland, learned what was going on from a German source. Riegner asked American diplomats in Switzerland to inform Rabbi Stephen S. Wise, one of America's most prominent Jewish leaders, of the mass murder plan. But the State Department, characteristically insensitive and influenced by anti-Semitism, decided not to inform Wise.

The rabbi nevertheless learned of Riegner's terrible message from Jewish leaders in Great Britain. He immediately approached Under Secretary of State Sumner Welles, who asked Wise to keep the information confidential until the government had time to verify it. Wise agreed and it was not until November 1942 that Welles authorized the release of Riegner's message.

Wise held a press conference on the evening of November 24, 1942. The next day's New York Times reported his news on its tenth page. Throughout the rest of the war, the Times and most other newspapers failed to give prominent and extensive coverage to the Holocaust. During World War I, the American press had published reports of German atrocities that subsequently turned out to be false. As a result, journalists during World War II tended to approach atrocity reports with caution." [http://www.history.com/topics/world-war- ii/american-response...](http://www.history.com/topics/world-war-ii/american- response-to-the-holocaust)

So I thank Edward Snowden, even more than I thank the rest.

<comment><USER3> > ...this knowledge was hidden from the vast majority of people who rely on TV > news...

They seem so much more enlightened now: [https://www.youtube.com/watch?v=XEVlyP4_11M&t=1336](https://www.youtube.com/watch?v=XEVlyP4_11M&t=1336)

> "Blow the lid off" doesn't mean the first person to reveal something...

Which is probably why you remember Riegner, yet give no mention of Leon Feiner.

> ...it refers to an explosion that can't be ignored.

Sure, or an artful twist to facilitate some hero worship.

<comment><USER4> Just credit where it's due, effective action is praiseworthy, and perhaps even more praiseworthy than relatively ineffective but noble actions. I don't understand your willingness to pick out one actor to knock. Why not at least praise them all, as I do? Because that's the difference between us, I praise them all and you want to knock one. Your blasts don't reveal much about your motive for that - do you perhaps want to argue against "Great Man Theory" and in favor of the "historical power of the masses", and somehow think any individual achievement anywhere in history would disprove the latter, or something?

The "artful twist" is built-in to not just that idiom but also "blow wide open." It's not a coincidence, it's central to the meaning.

<comment><USER3> > I don't understand your willingness to pick out one actor to knock.

I'm not, and I don't understand your necessity to pick out one above others.

> Why not at least praise them all.

Easy, now. Don't forget that was my argument.

> Your blasts don't ... (unintelligible) ... or something?

Ugh, no. Let's not digress, again...

> The "artful twist" is built-in to not just that idiom but also "blow wide > open." It's not a coincidence, it's central to the meaning.

That idiom is overused. That "artful twist" lost its impact a long time ago and "blow wide open" is just a variant. These days both are highly subjective (as illustrated above).

No, the impact from Snowden's actions are useful but no more explosive than what was, and is now becoming, known to those of us who were paying attention all along.

Which is the entire point of my original comment. Would you criticize the WSJ for not name dropping Trump in an article on political stupidity? Or did he blow that wide open, too?

<comment><USER4> Such condescension. But I don't think you've added anything. It's pretty clear on this page that we were amongst those paying attention, but didn't have much company. Thanks largely to Snowden, now we do.

<comment><USER3> After a litany of increasingly obtuse replies each growing further away from my original point to the extent that you actually attempted to turn my own argument against me, you better believe I'm going to be condescending.

Talk about not adding anything... Good grief, and good night.

<comment><USER4> Wow. Just wow.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER5> One of my relatives also said this "I have nothing to hide" BS, I replied by asking who they were talking about. Response: anyone but the criminals. Hmm, so I demanded to install a spy app on their phone to be able to monitor everything they do, where they are, record the conversations etc. They declined this offer, either implying that they perceive me as a criminal or they confirmed to being hypocritical about this reasoning. Also this reasoning implies that government agents and other officials are somehow immune from illegal or criminal activities.

Edit: my impression is that the general public somehow thinks that this data collected from mass surveillance is stored so securely that it can never be accessed by criminals. The only thing that could ever change this perception is some major breach of personal information followed by massive economic consequences, otherwise no one cares and politicians care too much about they own power that they will only expand surveillance efforts.

<comment><USER6> You just described the vast majority of my relatives. There is a lot of teaching for me to do...

</comment></comment><comment><USER7> The right way to think about privacy would be "is that privacy worthwhile to protect". There is no absolute in reality. Everything is a balance. And that balance is constantly shifting. We, human, are experts in handling shifting balance though. Do not be stupid (blindly trusting any party, including our government or google), but do not be paranoid either (worrying about potential crisis every day). Just live your life, and use common sense.

<comment><USER4> Go with the herd, don't go nuts look for cliffs upcoming. That could work. Or not.

</comment></comment><comment><USER8> The thing that worries me most about all the personal data the government has is not its using it to indict people. It is that it will take information that would be embarrassing to politicians and either use it to blackmail them, or hand it over to the opposition to use in an election (from what I understand, J. Edgar Hoover did that). If that is going on, then you don't really have a democracy any more.

</comment><comment><USER9> There are good points in the article, but I'm annoyed that they bring up the "lobster law" as so many articles about "absurd laws you might be breaking without knowing it" do. There's actually a very valid reason why possessing a small lobster is illegal -- immature lobsters haven't bred yet and if you want to have lobsters in the future you shouldn't harvest them.

<comment><USER10> The validity of the reasons doesn't come into this because the point isn't that "possessing lobsters being illegal is stupid". The pint is "lobster are illegal, who would've guessed, can you imagine having a lobster and ending up in jail?, and can you imagine how many other thousands of unknown laws are you breaking right now?, it's not possible to keep track of them all"

</comment></comment><comment><USER11> I would like to argue that the fourth amendment implicitly protects against legislation whose only method of enforcement would require unreasonable search.

If you cannot observe any direct effects of an action without searching the person who performed the action, then it _cannot_ be a criminal action, and theres a reasonable argument that this follows from the fourth amendment.

</comment><comment><USER6> Today we need to expand the title from "Surveillance" to "Privacy".

Privacy is integral to liberty. That is why this matters so much.

</comment><comment><USER12> Just tell politicians to submit to surveillance, after all, they should have nothing to hide.

<comment><USER13> Ideas like this sound sensible on the surface, but they have unfortunate side effects.

For example if politicians must disclose a lot of private information, only those who don't care about that will become politicians. And then since they are comfortable with it, they won't see any reason to control dissemination of others' private info.

(BTW I'm not criticizing you by saying this; I don't know a good fix).

But you know how everyone gets a blast of schadenfreude when that anti-drink- driving politician is pulled over for driving dangerously after having a few too many? Of course I laugh too, but I wonder if they have been so vocal about the issue because they know how dangerous it is. Whereas I, who drink little and don't even like to drive, probably wouldn't pay much attention to the issue if I became a politician.

<comment><USER12> I agree with you. But forcing lawmakers to give up their privacy too would at least, hopefully, theoretically, level the playing field..

Same way as we now at least hold them subject to the same laws involving money, taxes, and can use those laws to indict them.

<comment><USER13> > Same way as we now at least hold them subject to the same laws involving > money, taxes, and can use those laws to indict them.

Actually you can't really, and again for good reason, and again with unfortunate side effects as well.

Legislators have legislative immunity for good reason: otherwise people would use the courts to attack legislators for making decisions the attackers don't like (the emolument suit against Trump is avowedly similar[1], although he has a slightly different immunity). You can, after all, attack them just as well via the ballot box. In the US the FBI can still investigate crimes they suspect were committed by sitting legislators, and the house or senate can waive one of their members' immunity, so it kind of works.

At the opposite extreme, criminals run for office in India in order to take advantage of the legislative immunity from prosecution.

All this goes in the "some problems are simple to characterize but not simple to resolve" bucket. When people hear famous quotes from very smart people along the line of "if your explanation of your problem is complex, perhaps you don't really understand it" they often confuse that with "your solution must be simple too."

[1] I just heard an interview with one of the law professors bringing suit, Zephyr Teachout, who said she thought it would likely go nowhere but that it was worth making his life difficult. Even if you consider Trump an irredeemably corrupt mastermind, it is abuse of the legal system to use it when you know you'll fail. Not that this is the first time someone filed suit just to annoy someone else.</comment></comment></comment></comment>


The Blockchain Pipe Dream - <AUTHOR> https://www.project-syndicate.org/commentary/blockchain-technology-limited-applications-by-nouriel-roubini-and-preston-byrne-2018-03/english <comment><USER2> "Although we can be fairly sure that blockchain will not unseat TCP-IP..."

Do court intellectuals lack clue by nature or does their position as court intellectuals impose their lack of clue?</comment>


Goodbye MongoDB, Hello PostgreSQL - <AUTHOR> http://developer.olery.com/blog/goodbye-mongodb-hello-postgresql/ <comment><USER2> As a greying developer I am most amused by people discovering that 'old' technologies like SQL databases work really well.

The only useful piece of advice I can give a younger developer is... be careful when drinking the newtech koolaid.

And one more thing:

        star = Sequel.lit('*')

        User.select(:locale)
          .select_append { count(star).as(:amount) }
          .select_append { ((count(star) / sum(count(star)).over) * 100.0).as(:percentage) }
          .group(:locale)
          .order(Sequel.desc(:percentage))

just makes me want to cry. Learn SQL rather than wrapping it.

<comment><AUTHOR> It's not so much about not wanting to write/understand SQL (both are still very much required), but about composability. If you want to re-use bits of a SQL query written as a string literal your only option is string concatention or using some kind of string builder/template system. In both cases there's little validation of the query's correctness (syntax wise) until you actually run it.

While I agree that many ORMs go too far or even worse, not implement certain powerful features, Sequel (and similar tools I imagine) strikes a nice balance. The particular examples I gave on their own are not super useful, but we already have quite a few queries that are composed/re-used without having to concat strings.

So tl;dr: it's about composability, not being "lazy" or "ignorant" to SQL.

<comment><USER3> How often do you _really_ reuse bits of a SQL query? SQL is the language that an ORM will generate. The question is, how much work is done to avoid using SQL and is it worth it?

The only time that I've found ORMs useful for composing queries when I have to dynamically create a query at runtime based upon user input. And even in that case, today, I'd probably still just concatenate strings for a proper SQL statement.

<comment><USER4> I think there is a problem of tooling in SQL, which means it is difficult to reuse it.

I spent the other day writing a big query, joining about 10 tables. So usually I start by joining the two main ones. Check that gives me the results I want. Add in another table or conditions, check again. Repeat until all tables are joined into the query and conditions are added.

Then I noticed the results of my GROUP_CONCAT were not as I expected. I had a couple of suspect joins that I tried removing. Same problem. In these situations, it is often easier to go through the same process from scratch of adding in one table at a time and ensure that it is working.

We need some kind of unit test equivalent for SQL.

<comment><USER5> Mature relational databases support views, pre-built queries that developers can define and other developers can then use as "tables" in higher-level queries, allowing some code reuse. This has been around a long time.

</comment></comment></comment></comment></comment><comment><USER6> This post reflects an interesting technical narrative of companies switching off MongoDB to more traditional relational databases as they grow.

Importantly, I don't think that's an indictment of MongoDB. Instead, it highlights the key advantages of NoSQL: ease of use and rapid iteration.

When you're first working on a project, MongoDB is very easy to slap in. You don't even have to create tables/collections. As you iterate, you don't have to constantly be updating schemas as your product direction shifts. Thus, MongoDB is perfect for early startups.

As a company scales, of course it's warts start to show. The very attributes (like schemaless storage) which were an asset when just a few developers were hacking on the project become a drawback as the company grows in size. Reliability becomes more of a concern.

Naturally, people shift to a more traditional database. Personally, I'm almost ready to switch some of our core functionality from MongoDB to Postgres. This isn't an indictment of Mongo though.

<comment><USER7> Really, you can't spend the 10 minutes designing a table structure in a SQL database? And now you have to spend months re-inventing the wheel because you wanted an easy out?

This post reflects on developers being lazy, instead of doing it right the first time around. Oh no, you have to log in to the db and run a CREATE TABLE statement every few months when you need to scale. Cry some more.

And even then, 'lazy' is subjective. I don't find SQL that hard to implement. There's a library for practically every language you can think of. What's your excuse?

<comment><USER8> Actually, I think that's a great way to think about it: NoSQL is the "dynamic typing" of the database world.

Put another way, it's like "what? you couldn't spend 10 minutes declaring types everywhere?" \- yeah, it's less robust, yet dynamically typed languages remain popular.

My excuse: When I'm just past the mock stage, and still playing with what UI functionality should be, sometimes I just want to get some JSON persisted. I'm changing the shape of the data a lot, discovering the schema as I build. The persistence is frankly a technical nuisance I wish I didn't have to think about.

Sometimes, the result is "good enough" and I don't need to go through the ceremony of glueing in an ORM.

<comment><USER9> Your given excuse makes you sound more like a naive amateur instead of a pragmatic architect

<comment><USER10> Just in case GP cares: To me it did not sound like this at all. I think making prototypes and starting to build the thing is the key to getting to a good architecture.

</comment></comment></comment></comment></comment><comment><USER11> > Another way of handling this is defining a schema in your models. For > example, Mongoid, a popular MongoDB ODM for Ruby, lets you do just that. > However, when defining a schema using such tools one should wonder why they > aren't defining the schema in the database itself.

Bah. It's like they didn't know that schema-free data stores mean "there is no schema; different objects may have different fields". This is the whole point of MongoDB: you assume the responsibility of managing the schema. That's a "feature": you get greater flexibility by assuming more responsibility. Whether it's a useful feature, I won't say.

Their second reason is much more valid: MongoDB is not consistent/durable, all that. It's good for a cache, but not for long term data.

Here's a third reason I'd give against it as your primary data store: it's expensive. You have to keep your entire dataset in RAM, but that's not always necessary. My favorite example is from Foursquare. They had every single check-in ever in MongoDB in RAM. That's absolutely unnecessary, and quite silly to do so. Old check-ins are archived data. You don't need them. No user ever wants to know when/where they checked in three years ago. This is why at the time they were paying for 68 GB RAM Amazon boxes instead of 4-8 GB boxes. (I have no idea what they do now. I remember chatting with them on HN after a catastrophic out of memory failure when they filled up the entire 68 GB's).

<comment><USER12> We have a dataset much larger than RAM (600gb vs 60gb RAM). It's the working set that really matters. Accessing data outside the working set can be slow - unless you have SSDs :)

Regarding consistent/durable: during the past four years, we've not had any problems on this front that weren't caused by us in. We've had an issue that was a misconfiguration on our part where we allowed writes to a server because we pulled it out of a replica set. We also ran out of space on the logging volume once and that caused downtime - but, we didn't have log rotation or anything setup.

In general, we've found the failover very reliable and new primaries have come online without any problems.

That said, schemaless is both a blessing and a curse. Now that Postgres and MySQL have online alter built-in I'd possibly choose one of them if we were starting everything again.

<comment><USER11> Well, SSD's aren't nearly as fast as RAM, even today.

I think we are in agreement about the working data vs all data. I am saying that in most applications your working dataset is much smaller than your total data set. So why pay for hardware capable of holding your entire dataset in RAM when you don't need it?

I am surprised you are able to do this with Mongo. Last I checked, it simply did not handle this case, and started failing miserably if it was not able to fit all data into RAM.

Re: durability: I am not talking about the server going down, coming back up. I am talking about whether there is an fsync() when writing data. Set up a test case where you are writing data very rapidly to MongoDB, then pull the plug on the box it's running on. It'll come back up, but the data it told you it just wrote won't be there because it didn't checkpoint. Did you check that all your writes succeeded when you had node failures in your cluster? Most applications don't have the machinery to do this because generally the state necessary to check this is stored in the database, yet it's the database you are testing. The only way to test this is to also write logs (also atomically), and then verify DB data against logs. Or, just use a database that guarantees durability.

Re: consistency: MongoDB doesn't support transactions [1]. That's enough to exclude it from a large number of applications. Anything to do with money, for example, is out since you really don't want double spending to be a thing.

[1] [http://docs.mongodb.org/manual/core/write-operations- atomici...](http://docs.mongodb.org/manual/core/write-operations-atomicity/)

<comment><USER12> Fair points - we don't allow failed servers back into replica sets and rely on writing to multiple nodes instead of the disk as source of truth.

This may not suit everyone and absolutely does not suit financial transactions. You can bend Mongo to do it using additional collections and money movement logs... but, why bother when it's simpler to use MySQL/Postgres?

</comment></comment></comment></comment><comment><USER13> There is a mistake in the article, due to the OP not knowing an arguably basic notion about MySQL.

> when defining a field as int(11) you can just happily insert textual data > and MySQL will try to convert it.

this is dependent on the SQL Mode, which is quite flexible. for example, the STRICT_ALL_TABLES will prevent strings to be inserted in INT fields:

mysql> create table example ( `number` int(11) not null );

mysql> insert into example (number) values ('wat'); Query OK, 1 row affected, 1 warning (0,00 sec)

mysql> SET sql_mode = 'STRICT_ALL_TABLES';

mysql> insert into example (number) values ('wat'); ERROR 1366 (HY000): Incorrect integer value: 'wat' for column 'number' at row 1

There are certainly advantages in choosing PostgreSQL over MySQL... this is just not one of them :-)

<comment><USER14> Personally, I would not use a database which is 'lax' by default instead of 'strict'. What other choices have they made which I need to learn OR it will bite me big in Production?

<comment><USER13> It's not so easy.

MySQL has a large legacy of being used as a very-immediate-although-somewhat- toy database at its roots.

For example, in absolute terms, I would find much more troubling the usage of non-transactional tables, justified by meaningless microbenchmarks, which has been somewhat common for some time.

Nowadays MySQL is definitely reliable, and it has a much more expert surrounding culture than the past, so if a person/company is willing to put time and knowledge, it's a reasonable choice.

I don't find automatic conversion so damaging that people should stay away just because it's default.

When you reach some level, you definitely need to have a relatively intimate knowledge of your tools, and at such point, one is far from the "defaults".

</comment></comment></comment><comment><USER15> _In the future we might also move our Rails applications over to Sequel, but considering Rails is so tightly coupled to ActiveRecord we're not entirely sure yet if this is worth the time and effort._

Actually, with modern versions of Rails, using Sequel in place of ActiveRecord isn't bad at all. Nothing in Rails is really tied to ActiveRecord anymore. There are dependencies on ActiveModel, but you can easily make Sequel::Model objects conform to this interface. Sequel-rails helps with most of that: [https://github.com/TalentBox/sequel- rails](https://github.com/TalentBox/sequel-rails)

It's really just the migration that's difficult, as that's tough to do piecemeal and requires good test coverage.

Here's a recent side project of mine that I switched from ActiveRecord to Sequel pretty quickly once I remembered how extremely limiting the querying capabilities of ActiveRecord can be: [https://github.com/bgentry/portfolio- api](https://github.com/bgentry/portfolio-api)

And if you really want to keep the option for schemaless data storage, Postgres can now do that with better performance than MongoDB, while keeping full indexing capabilities: [http://blogs.enterprisedb.com/2014/09/24/postgres- outperform...](http://blogs.enterprisedb.com/2014/09/24/postgres-outperforms- mongodb-and-ushers-in-new-developer-reality/)

<comment><AUTHOR> While certainly possible we simply haven't really evaluated it yet in depth. In between releasing a bunch of upcoming features and upgrading Rails from 3.2 to 4.2 I'd rather wait with _also_ moving from ActiveRecord to Sequel for the time being.

<comment><USER15> yes, by all means upgrade yourself to the latest release of Rails before attempting to go "off the rails" :)

</comment></comment></comment><comment><USER16> The article is sort of weird, why Document, no sql, no schema DB to be a schema DB? One fundamental fact that you liked MongoDB in early days (first 5 years) because it was damn easy to handle anything you throw at, you learned to build the business around it. In 5 years, you learned enough, business model matured, not much changes in the data model. Now development progress is matured, you have got team to work on, your worries relies on consistency. You might want to back to SQL. Now you know what schema/table you need, string length, data type, constraint, relationship etc, because you learned that in 5 years time. You don't need to deal with new developers screwing the Document DB (like storing Object ID as string in few places)

Developers, who reads the article beware of reality, you can look back 5 years now, not looking forward 5 years ahead. Don't waste time in dealing with database columns and schemas, instead build the business faster, I found Mongo DB or any Document DB is good fit for agility.

1000's of business move to Document DB because it is schema-less.

While working on Document DB, you should be master in writing stand-alone scripts in Python/Perl/Ruby to run every-time you break the structure, or fix inconsistency.

edit: fixed typo errors

</comment><comment><USER17> Is there anybody here who has run MongoDB at moderate scale with good results?

As in a few terabytes of data, >10k ops/second territory.

I've been really disappointed with its reliability and performance in situations where I've been around that.

<comment><USER18> I've been using mongo on 39M+ records (tracking financial tick data) across 73 assets (collections) and my queries take anywhere between 2-3mins depending on complexity.

You can always run db.currentOp() in the mongo shell to see what process is taking forever as well.

Let me clarify why though, there methods of optimizing a query by adding another field, but since I have to traverse my records with the sort() cursor my queries take that long.

<comment><USER19> Jesus Christ!

I remember querying databases with 100 tables and millions of records in foxpro a century ago and it took less than a second.

What has happened to the world while I was in cryogenic state? Take me back to the nitrogen pool!

</comment></comment></comment><comment><USER20> I had a business based on another company's API years ago, and they started changing a lot of their systems to work with mongo. The data lost a few useful points, like IDs and the speed and reliability didn't seem to be improved. My thought at the time it was, how about you guys just use postgresql? My comment on the mailing list about that seemed to be taken as an insult or naïveté. However, sure enough two or three years later they said they had all sorts of problems with Mongo and were switching to… Postgres.

</comment><comment><USER21> The author's assertion that "Another problem with MySQL is that any table modification (e.g. adding a column) will result in the table being locked for both reading and writing. This means that any operation using such a table will have to wait until the modification has completed." is no longer correct as of Mysql 5.6:

[http://dev.mysql.com/doc/refman/5.7/en/innodb-create- index-o...](http://dev.mysql.com/doc/refman/5.7/en/innodb-create-index- overview.html)

If you specify ALGORITHM=INPLACE,LOCK=NONE you can alter table without blocking reads and writes. We have used this method successfully in Amazon RDS when updating schemas.

<comment><USER22> It's not exactly a common operation either, so basing the choice of rdbms on it seems a bit arbitrary.

<comment><USER23> It happens a couple of times every month in our production databases. If it required locking tables we would have to schedule downtimes during the early mornings every time this happens would would have been a pain.

</comment></comment></comment><comment><USER24> As soon as you have more then 1 app talking to your database its time to either create a common library that talks to your database that can be shared or wrap it in a service. This has nothing to do with NoSQL vs SQL. It has to do with architecture composition.

Plus your problem with checking if a field exists is because Ruby doesn't support property attributes. This is easily solved in C# using attributes. Which you can fake in Ruby. [http://stackoverflow.com/questions/1085070/how- do-i-fake-c-s...](http://stackoverflow.com/questions/1085070/how-do-i-fake-c- sharp-style-attributes-in-ruby)

Data consistency is what your models are for. No data should be inserted before being assigned to a model.

Again it sounds like you needed a shared library or just a service wrapped around your database.

</comment><comment><USER25> Something that I need to call out here. In talking about how to handle the lack of fixed schemas within Mongo, the OP refers to using if/else blocks to get around field name changes. This shows a severe lack of understanding of how to work with Mongo at such a fundamental level. If using Mongoid (which he is) you simply need to run `Post.all.rename(title: :post_title)` to do the schema change. This may seem like a nitpick to some people, but honestly if that concept was lost on the developer then its clear that there is little to no credibility to be had in the reporting of all the other Mongo issues.

I'm glad that the OP is growing as a developer and starting to understand the merits of SQL and why having defined schemas is important. SQL is awesome and its not going to be replaced by NoSQL (at least not fully). However please make sure that you are building the foundation of your understanding by fully grasping the technologies that you are already using.

</comment><comment><USER26> > ... we value the following: > Consistency

With all due respect, you should have had this list before selecting MongoDB. Consistency, as in "eventual consistency" and also in the way that you describe it is better supported outside the NoSQL group.

Also, I don't fully agree with the "schemaless" discussion. The moment you think about your data, you build a schema in your head, which, then, is translated into code etc. It's about how far you take this rigid model. Maybe you should see it as "flexible schema".

</comment><comment><USER27> So let me ask a question. What should I use when I do need a schemaless database? Is NoSQL never the answer? I've got a project that needs to allow clients to create registration forms for different events that my company hosts. A lot of the registration data will have a defined shema ex: name, email, address. I feel like that stuff should go in a RDMS, but all the event specific stuff needs to be schemaless. I know I can do custom key/value tables in a RDMS, but that doesn't feel right either. Is MongoDB useless as a database, or are people being bitten for thinking it's a silver bullet and throwing it at every problem?

<comment><USER28> Riak is amazing and actually scales. But I would only use Riak for high-volume data storage (similar to S3).

FoundationDB looks great, I haven't used it yet but they appear to have their heads on right.

PostgreSQL, the newer versions, have indexable BJSON data types so you can get the same exact behavior from Postgres as you do from Mongo but with a true RDBMS along with it, a dependable storage engine, etc...

Postgres is harder to scale horizontally though - if you have really high- volume data writes, you should be using something else for that.

I typically use PostgreSQL for all of my highly "structured" data and Riak for high-volume and "flatter" data (Postgres also often serves as an index _into_ those objects).

<comment><USER27> Awesome, Thanks for the reply. My use cause is barely out of the "toy app" range. We only do a handful of events each year and they only draw around 100 attendants. We're talking a very small amount of data. When you do the PostgreSQL and Riak combo is it ever on the same/related dataset. What are you using at the application layer? I'm building this in rails and I feel like it would be better to store the structured fields in MySQL and the variant data in something NoSQL. But I haven't read much into using active record with two different persistence layers. That's interesting about PostgreSQL with indexable BJSON. I've only ever used MySQL, I really need to check out Postgres.

<comment><USER28> It's easier to go from something more highly structured to something looser. Start with a relational db and let it grow then pay attention to what data gives you the most scale pain and try to move that out to Riak / Cassandra / etc...

Don't prematurely scale, just pay attention to your metrics, scale vertically first, then tackle the very specific pain points.

</comment></comment></comment></comment><comment><USER29> while i've not spent much time with the "nosql" products, its mostly because i've decided to stay SQL. The bottom line is that most humans think of data the same way SQL thinks about data. When you talk to people outside of engineering (i.e. mgmt or customers), they expect the product to be able to do things that SQL does naturally (normalized data for easy changes to things like 'user name', join and sort based on a variety of cross cutting properties, etc). You can always make both do the same thing eventually, but one will fight you more than the other....

</comment><comment><USER30> I did a similar migration last year. Loving postgres.

</comment><comment><USER9> This post outlines some of the most expensive parts of deploying, operating, and maintaining an application that operates on "NoSQL" databases like Mongo, and in my experience, DynamoDB:

1) Implicit Schemas. We avoid this completely by doing production migrations and re-indexes on every new field we add. Its expensive, and we have to write/test/run scripts in production. At times I wish I could just write a Rails migration on an RDS instance and call it a day. N-1 compatibility isn't hard to accomplish with good code reviews.

2) Search. Want to search and join like in the old days? Good luck. Using a NoSQL DB as the primary authoritative store of record is great, but you'll need a secondary indexes for any searches you want to do. If you need to look up an object on a new field (or even one that already exists in all of your data) if you haven't built an index for it you will have to.

3) Serving Clients. Because of the schemaless-blobby nature of writing clients, things get real messy the minute you have multiple services or applications reading or writing to a DB. To get around this, you have to put a service in front of it and serve the data from some RPC technology, which is an extra step and requires more development and maintenance.

4) Administration. There aren't a long history yet of robust toolsets and "science" behind different schemaless NoSQL databases. Meaning which one you choose has a huge impact on your ability to fine tune it, debug consistency or other expectation issues, and do things like proper failover, backups, restores, etc. Knowledge between similar NoSQL DBs doesn't transfer as well so your mileage will vary more so than on SQL databases (MySQL vs PostgresSQL, for instance).

NoSQL has its place and purpose, but it is rarely as the "one database that rules them all" that many businesses end up with. I'd be interested in counter stories.

</comment><comment><USER31> Yes if you have one MongoDB database that uses `title` and another one that uses `post_title` then you have to adjust your code for that.

Guess what. Same thing applies to SQL.

<comment><USER32> In SQL you can't have a table whose title field is called either `title` or `post_title` depending on which record you're looking at.

<comment><USER33> You easily could have this in SQL if you designed a poor schema or failed to migrate data from the previous field name.

</comment></comment></comment><comment><USER34> I haven't used MongoDB in production (the comments regarding reliability have been around for awhile), but playing around with it, I do like the json format and query structure.

The issue I have with SQL (MS sql in the case of work) is the amount of cleverness involved in some queries I have seen. Among the old timers, it seems almost a badge of honer to develop the longest, most clever SQL query that does everything in one step. Inevitably, there are problems, and people have trouble figuring out why because they can't debug parts of the statement in isolation. In this case, it's as if they have written an entire program in one line and can't test parts in isolation. No doubt this is abuse of the language... and admittedly my SQL skills are not world class, but I keep scratching my head and wondering why they do this to themselves. If a query is so complex you can't tell what it is doing, and it misbehaves, perhaps you would have been better off with some smaller queries you join in the program... nothing against joins... one or two or three of them... but really....

I keep asking myself if I'm missing something, but I sort of doubt it.

<comment><USER35> That is then not a problem of the SQL-Language, it is more a problem of the database design.

When you need really complex queries, you should think about a new db structure.

<comment><USER34> The largest database is vendor provided so we are stuck (for now).

Thanks for the insight.

</comment></comment></comment><comment><USER36> I switched a new product in planning from MongoDB to PostgreSql 9.4. With the new JSONB columns you get the best of both worlds.

</comment><comment><USER37> Interesting to read both the article and comments. It's cool to hate Mongo nowadays but we use it successfully (for PEPS, open source: [https://github.com/MLstate/PEPS](https://github.com/MLstate/PEPS)).

The version of MongoDb used is not mentioned in the article. The changes between versions is a problem with Mongo but it improved much recently.

One problem clearly identified is the lack of model. There are solutions to this. For instance, with Opa ([http://opalang.org](http://opalang.org)) we use strong static typing to generate a database model and guarantee that the whole applications sticks to it. That leaves out model changes, but there are solutions for that.

Also, there is no need to have the whole collections in RAM, but clearly enough RAM for the "working set" helps.

In the end, Mongo is no magic so do SQL databases which have their share of problems too.

</comment><comment><USER38> I made a little bet with myself when I saw the topic of this post that a good 80% or more of the top comments would be of the "Shouldn't jump on the MongoDB/NoSQL bandwagon, nothing wrong with SQL" variety combined with various arguments to support that position or arguments against NoSQL in general or MongoDB specifically. I won.

Here's the thing; anyone that claims one is superior over the other without adding a specific use case or context is doing something akin to religious preaching. Databases are tools. Pick the tool you need for the job. If you don't understand exactly what a hammer is for and how it does what it does put effort into gaining that understanding. If after that research you think the tool has flaws that are important to you then don't use it.

</comment><comment><USER39> There is only one NoSQL database satisfies all following requirements: 1\. easy to use 2\. reliable 3\. transaction support 4\. easy to scale 5\. eays to build indexes

It is BigTable.

Generally, if your data is not very large, you should use a SQL database. NoSQL is mainly used for easy scaling, not for its schema-less feature.

</comment><comment><USER40> Interesting article, did you consider any of the newer NoSQL solutions that offer a data model more similar to MongoDB but more, often configurable, consistency guarantees (like ArangoDB or RethinkDB)? I could imagine that the effort of a migration would have been considerably smaller...

</comment><comment><USER41> As the post explained, one of the big problems of migration off of MongoDB is finding out the "effective" schema of the data, and then migrating it (according to that schema). By "effective" I mean the resulting structure of the data stored in the database.

I think it would very helpful to use here ToroDB ([https://github.com/torodb/torodb](https://github.com/torodb/torodb)). While being MongoDB-compatible, it stores data structured into PostgreSQL tables, automatically identifying the schema of the data. Then, just by looking at the created tables, you very easily have the schema (and the data migrated to that schema). It would make migration easier.

Disclaimer: I am a ToroDB developer

</comment><comment><USER42> The problem of schemaless database mentioned in the article is then replaced with painful migrations.

The trend to go back to SQL databases now is fueled by the "same" poor reasons why people were going for NoSQL databases. You can't have everything...

<comment><USER43> I have found that the more schema you have, i.e. the more structure you have for your data, the easier migrations become, because there are no surprises.

</comment></comment><comment><USER44> Diving in a cesspit makes one appreciate clean air.

</comment><comment><USER45> It's very interesting to see the amount of negative articles about MongoDB, and at the same time see that its very popular on job boards. I'd honestly like to read an article about a success case

<comment><USER46> The successes tend to be quiet. And folks who are happy don't take to the internet to defend their choices.

I've been on two straight projects where Mongo has been fine, no better or worse than an SQL database, but certainly nowhere as bad as one would assume reading all the negativity around here. I have experience with it being stable in two different contexts in production (social games, a very common use case, being one). A search for "social games mongo" should show you some success stories.

The thing that makes it a good choice for me when starting a new project is that you don't really think about it, especially if you're following lean startup methodology where the goal is to get a product into the hands of the users as quickly as possible & you anticipate changes to your models. Mongo is pretty fantastic for this. And when things stabilize, moving data really isn't that bad or expensive if you modeled based on this assumption (which you should). And now that PostgreSQL has a pretty good JSON datatype, it presents a pretty painless path if and when you outgrow Mongo.

</comment></comment><comment><USER47> A related read, titled Why You Should Never Use MongoDB: [https://news.ycombinator.com/item?id=6712703](https://news.ycombinator.com/item?id=6712703)

</comment><comment><USER48> Beware of "The Problem Of Schemaless" can occur with a relational DB when you start defining columns as nullable. You end up with the same problem.

<comment><USER49> I wouldn't call it the same problem. Something nullable is similar to an Optional type (i.e. in scala Option[String] would map to a nullable varchar). That is not the same as being schemaless, you at least still have types that are being enforced, even if they might be empty.

I would agree that having too many nullable fields could indicate a problem with the schema, i.e. you should break things up into more tables.

<comment><USER48> Agree. I'll rephrase to "similar" problem.

</comment></comment></comment><comment><USER50> I also keep reading about how bad is MongoDB for lots of projects. I don't think MongoDB is worthless, but from my experience I also have concerns about it.

If so many people migrate to other technologies, why is MongoDB still so popular? Have a look at [http://db-engines.com/en/ranking_trend](http://db- engines.com/en/ranking_trend), MongoDB has just passed PostgreSQL

</comment><comment><USER51> They don't write if they looked at other databases. I think there are a lot of alternatives that might have worked better for them than MongoDb and even Mongo 3.0 might have worked better. Regarding the schemalessness for simple new properties they should probably have fixed that in the ORM layer, it can add default values if needed. That something is null can actually happen in sql also.

</comment><comment><USER52> > For example, when defining a field as int(11) you can just happily insert > textual data...

My eyes practically fell out of my head when reading this.

</comment><comment><USER53> The section, "The Problem with Schemaless," blames the technology instead of whoever put the data in there in the first place.

If the code has to handle both page.title and page_title, this is a feature of using a schemaless technology.

Also, lots of the issues the author had with MongoDB are also to be found in MySQL, e.g. taking hours to recover from a corrupt database/datastore.

<comment><USER54> It's just that if you have a sizable team (or worse over a long time with low overlap) working on a product, you intrinsically get an accumulation of duplicates, deprecated, or both "keys" in your schema-less schema.

With a DB that enforces a schema, the overhead of modifying the schema tends to moderate that nature.

I once worked in a lab (Ph.D. students are atrocious programmers BTW) where I introduced a schema-less store (cheesy K/V store) to handle some mundane metadata caching on some medical imaging. It was intended to store 4-5 attributes per PK, I never touched it after setting it up for what I needed but showed it to colleagues.

Fast forward 10 years and there were something like 3000 attributes defined. Several hundreds of which were serialized blobs. Huge amount of overlap between the different attributes.

Almost all that because people didn't know what was already in there so they just did their own thing.

</comment></comment><comment><USER55> I'm also moving back to Postgres. Postgres was the first DB I used when I learned to code PHP. MySQL came later. Just recently, around a year or two ago, I got introduced to Mongo and all its surrounding hype. Mongo is good for small small apps that need no references and have a low requirement for speed and data integrity.

</comment><comment><USER56> I've been thinking of the same changes for a few months. We are running on MongoDB with a few M entries, and around 10GB of data. We use Scala and ReactiveMongo on the application side. So, my main fear is we're going to have a lot of overhead when trying to migrate to something like "ReactivePostgres".

</comment><comment><USER57> It's very tiring to read things like this. And that is not meant as support for Mongo, quite the contrary in fact.

<comment><USER58> It's interesting that if you search for MongoDB vs Postgres, you won't find _any_ articles suggesting you should go from P to M.

Supposedly WiredTiger for M 3.0 will be more everything, including faster and better.

We'll see.

</comment></comment><comment><USER59> The concept of moving from one database technology to another, especially something like NoSQL to Postgres, sounds like a huge task.

Aside from redesigning the schema/model and changing all the code, what about new backup/restore procedures? Scaling and performance best practices? Did you need to hire a postgres expert?

<comment><AUTHOR> We didn't hire any experts, instead we educated ourselves on the matter. For example, one of the first steps we took was to run some rough benchmarks on Pg to see how it behaved compared to MySQL ([https://github.com/olery/rds- shootout](https://github.com/olery/rds-shootout)). Followed by this was mainly discussing PostgreSQL vs MySQL with those who used either one (or both) in production for a somewhat serious workload.

Backup/restoring is handled by Amazon RDS in our case, in the past we had a custom backup system for Mongo that backed data up to Amazon S3.

<comment><USER60> Will you publish the results of the benchmark?

</comment></comment></comment><comment><USER61> You probably had a replica set and maybe an off-site replica. What are you using now, a single PostgreSQL instance, a master-slave cluster or any other distributed setup? If positive, which one of the many psql distributed technologies are you using? Thanks.

<comment><AUTHOR> Before (Mongo): 1 primary, 2 secondaries, 2 arbiters Now (Postgres): 1 primary

Most of our applications require write access in some shape or form, so at least the default replication of Amazon RDS doesn't cut it. Besides that we don't really need it so far, don't see the need for it in the coming months either.

<comment><USER62> One problem I have with postgres right now is that whenever I want to upgrade the cluster (say from 9.1 to 9.3) there's downtime while the cluster is upgraded.

Not even vanilla replication helps, right now. I think I'd have to use something like slony to replicate between different versions of postgresql, but I never tried it.

How does upgrading postgresql versions work in RDS?

</comment></comment></comment><comment><USER63> Wonder if they evaluated rethinkdb?

<comment><AUTHOR> I briefly looked at it, but seeing how young it was (and still is) I'd prefer not to bet any money on it for now, I'd rather use something tried and proven.

</comment></comment><comment><USER64> I feel like I'm missing something. So many comments in the article and here along the lines of, "just issues a warning and not an error."

Am I the only one here who's thinking that this is correct behavior on the part of the DBMS? Three result codes from an operation: 1) everything is okay, 2) I'm sorry Dave, I can't do that (error) and 3) Okay, if you insist, but I'm going to change your data to make it work.

Am I the only one who thinks: the programmer should be aware of and respond appropriately to ALL THREE, not just 1 and 2.? That anything else is just laziness?

Or is that just me? Am I missing some subtle consideration here that results in my thought process being naive? If I'm being naive I do want to understand what I'm missing, because getting schemas right and having my code react when improper data types are being used is sometimes a pain, ORM or not. But I've always thought it was the right thing to do.

<comment><USER65> Personally I feel like 3 should not be an option. Either things are okay or not okay, I don't like the "sort of okay maybe" option because that implies lots of fuzzy definitions and boundaries that the developer just has to learn.

Schema says int? Give it an int or fail. Not "it's sort of okay if you give it a thing which could be coerced into an int according to the database's ideas of coercability".

</comment></comment><comment><USER66> Thank god for return of sanity.

</comment><comment><USER67> Who doesn't want NoSQL with ACID properties? Can anyone tell me what is superior on MongoDB compared to PostgreSQL?

If anybody is going to talk about scalability, there are many good alternatives to scale PostgreSQL too.

</comment><comment><USER68> Not that im excusing mysql, but what I do when adding columns to a +10mil row tables is to create a new table with the new column and then insert into newtable from oldtable.

That changes those few hours into few seconds.

</comment><comment><USER40> For some additional thoughts about schemaless vs. schema-enforcing datastores see

      https://gist.github.com/neunhoef/9b6749089775e472d44c

</comment><comment><USER69> What system is he using to graph his queries response times?

<comment><USER70> New Relic.

</comment></comment><comment><USER71> Care to share your experience in terms of performance of Postgresql's JSONB vs Mongo? for both selects/inserts? and how about nested search queries?

<comment><USER72> In general, NoSQL databases are good for storing unstructured data. They start to fall apart when you want to query that unstructured data. You will hate yourself if you want to query something that is nested inside something, especially if one of the nesting levels is of the array JSON type. This is because indexes are required (just like in a normal database). Many times, Map-Reduces are required since the queries don't run in reasonable runtime. At some point, because you are storing the entire JSON document, relational databases start winning the space game (they don't store the name of the column with every document, like NoSQL does).

</comment></comment><comment><USER73> Am I alone in thinking that if a programmer writes a code that allows a string to be sent to an integer field in the database the issue is not with the DBMS?

<comment><USER22> Yeah. And since they use ActiveRecord, how would they even go about doing that in the first place? Over all the years I've used MySql, that has never been an issue for me. Sure it has its quirks, but show me a mature piece of software that doesn't.

</comment></comment><comment><USER74> This maybe related:

[https://news.ycombinator.com/item?id=6712703](https://news.ycombinator.com/item?id=6712703)

</comment><comment><USER75> There are ever more use cases for which relational data models are a poor fit.

For everything else, there are relational DBMS.

</comment><comment><USER76> Do anyone recognize the application used for the metrics screenshots?

<comment><USER77> New Relic

<comment><AUTHOR> Correct, this is taken from New Relic's APM service.

</comment></comment></comment><comment><USER78> I think strict mode makes the warning an error

</comment><comment><USER79> My aversion to NoSQLs is derived from the readiness with which uninformed people dive into them, integrate them with their products, and create a web of complexity around something that should ideally be boring and reliable: the database.

There are so many things that I've heard you "can't do in SQL" that are false, at least pertaining to Postgres. Semi-structured data, full-text search, "web scale" programming, geographical indexes... all of these things, people say "you can't do in a relational database" and that's not true. Postgres is fucking powerful and can do a lot, very well.

Arguably, PostgreSQL isn't always a "relational database". You can _use_ it as a key-value store. It just happens that you often _want_ relational logic in a multi-purpose, long-lived data store. The relational database seems to be an attractor; the requirements that accrue to a typical in-house "we can do it better"/NIH non-relational database often converge on it.

NoSQLs have their place at very large scale (100+ TB) and there are plenty of specialized reasons to use alternative databases-- I doubt that Postgres's full-text search is competitive with Elasticsearch-- but I feel like most of the anti-SQL sentiment is against _the language_. And sure, it's an ugly and outmoded language, but the database is one place where I'd rather have an ugly language and rock-solid tech than the other way around.

Finally, fuck ORMs.

<comment><USER80> Agree with all of your points. The thing that perhaps annoys me the most is that people seem to assume that nosql is magically fast and that sql is slow... Sure, a key-value pair lookup runs like shit off a shovel, but as soon as you want to run anything more complex than that, it's likely faster with a relational database and a few indexes...

Usually the same people that think running an application on a cloud platform magically makes it fast, when in reality a VPS would be cheaper and faster.

<comment><USER81> NoSQL is not a well-defined term - there are huge differences between various solutions all commonly called "NoSQL", so the comparison to NoSQL is bogus. There are different NoSQL things designed with different use-cases in mind. On one side you'll have things like MongoDB, which is easy to set up, but doesn't really scale once your dataset grows out of memory, on the other side things like Cassandra which are unmatched by any RDBMS in terms of performance, scalability and availability on real-time, transactional workloads. Of course, your RDBMS of choice might be better at complex JOINs than Cassandra (which doesn't have JOINS at all, btw), but will it stand the chance at performace competition with Apache Spark or Hadoop? I don't think so.

</comment></comment></comment><comment><USER82> Isn't this a little premature to announce a victory? After all, they've been using MongoDB for 5 years and it delivered for most of the time. Who knows if this new shiny SQL thing will be sufficient for next 5 years? I'd say it's quite likely after few rounds of excited development the database will be slow and crappy again ;)

</comment><comment><USER83> No comment.. (facepalm) and bad PR.

</comment><comment><USER84> MongoDB was never about its benchmarking, scaling or sharding abilities. It was about two things:

#1: No schema. If you're prototyping, schemas just slow you down. Maybe that changes as your application matures.

#2: Great drivers. The amount of pain to get a basic CRUD app running with a SQL backend is just too high: push the data from the browser to the server in JSON or XML, validate, convert into database schema, sanitize using prepared statements or stored procedures, and send it off. Compare that with MongoDB: _it 's just JSON™_.

The one thing that killed MongoDB was lack of reliability. Even when you're a startup, you may not lose data. I think MongoDB could have easily dropped 50% of performance & scalability for data security and it would be well off now. It was never intended to be the safe, sane choice for big enterprises.</comment>


Ask HN: Has it become harder to rank on the front page of HN? - <AUTHOR> I've seen a lot less personal projects/ebooks on the main page. Is there a reason for this? <comment><USER2> You're hitting a confirmation bias.

<comment><USER3> If you mean _Be the change you want to see_ then OP simply has to post to HN with a strong bias towards his/her topic of interest?

I can't see that by merely choosing a topic you are passionate about, that it gets more up-votes, but this is the tactic I do see because by being passionate, by virtue, you post more, and so more upvotes are guaranteed?

<comment><USER4> I think he might be referring to the fact that if you can't rank on HN you are prone to believe it is getting harder to do so.</comment></comment></comment>


Electrical impedance made easy - Part 1 - <AUTHOR> http://benkrasnow.blogspot.com/2011/06/tutorial-electrical-impedance-made-easy.html

<comment><USER2> Put simply, impedance is just complex resistance. We get it by taking the Laplace transform of the differential equation that defines the relationship between current and voltage in a capacitor or inductor.

        I = C dV/dt
        Laplace(dV/dt) = V*s
        I = C * V * s
        V / I = 1 / (C * s)

        V = L dI/dt
        Laplace(dI/dt) = I*s
        V / I = L * s

Another way to look at this: since this is a linear system, we know its eigenvalues are complex exponentials, i.e., their real and imaginary parts are sinusoids of any frequency (by the Euler identity). So we can say

        V = Vm * exp(j*omega*t) -- using j = sqrt(-1) to avoid confusion with current
        dV/dt = Vm * j * omega * exp(j*omega*t)
              = j * omega * V
        V / I = 1 / (C * j * omega)
        | V / I | = 1 / (C * omega)
        arg(V/I) = -pi/2 -- voltage lags current by 90 degrees

By a similar argument,

        | V / I | = L * omega
        arg(V/I) = pi/2

We can see that as expected, s=j*omega.

<comment><USER3> All I can say is that if this is the "simple" explanation to you, you and I think in very different ways. :)

To me, it's _much_ easier to understand the visual where impedance is the hypotenuse of a triangle whose legs are resistance and reactance.

<comment><USER4> That doesn't really _explain_ impedance. First of all, you need to know what resistance and reactance are to get that explanation, and second it doesn't explain why it's useful or how to do calculations with impedance.

On the other hand, kwantam's explanation is easy and clear, provided you know what a Laplace transform is (which you're going to need to understand reactance anyway). If you set up the system of differential equations for a linear circuit, how would you go about solving these equations? You take the Laplace transform, and you get the equivalent algebraic equations of the form V = ZI for each element. The numbers Z that appear are the impedances.

From a mathematical viewpoint, impedance is a simpler and more natural concept than resistance and reactance, since it appears naturally when you solve linear circuits.

<comment><USER3> > That doesn't really explain impedance. First of all, you need to know what > resistance and reactance are to get that explanation, and second it doesn't > explain why it's useful or how to do calculations with impedance.

It's amazing to me that we can see this in such opposite ways. To me kwantam's post doesn't explain impedance at all, it just formally defines it. It doesn't even begin to explain to me what it measures or how it applies to an actual circuit. What components of a circuit have an impedance? If I increased the impedance, what would happen? How would I even do that? kwantam's post doesn't begin to answer these questions.

I don't get any intuition at all for what this concept _actually means_ from a formal definition. That's why teaching methods that start with formalisms have always frustrated me. I bet you could dress up a really simple concept in a formalism that would be totally indecipherable. I actually think this would be fun to do sometime: describe some mundane concept in a really formal way and then ask a bunch of mathematical people "what am I describing?" And it turns out to be something boring like the rules for driving through an intersection.

I guess my point is, formalisms seem obvious once you understand the concept intuitively, but are a terrible way to gain that intuition. The basic idea of resistance is simple (a resistor restricts the flow of electricity), reactance was new to me but I got a rough idea that it similarly resists the flow of electricity in an AC circuit and is a function of the frequency.

So if I know that the impedance is a right-triangle combination of those two things, I've got a great start for getting some intuition: I know that anything that has resistance _or_ reactance also has impedance, and something that has _both_ has even more impedance, but in a DC circuit where the frequency is zero you'd get some edge condition where a non-zero reactance would, let's see, act as infinite resistance, I think? The formal definition doesn't plant any of these paths of understanding or inquiry in my brain.

<comment><USER5> I understand the Laplace transform ideas above (used to TA the class where it's taught) and I agree with your that they are more definitional than explanatory.

To provide actual explanatory power, you'd have to explain what the reactive portion ("imaginary part" in the phasor notation) of the impedance is actually doing in real terms.

What it's doing is to cause the current to lead or lag the voltage in the AC signal. In a purely resistive circuit, current and voltage vary exactly the same way at every instant.

You can actually go a long way with this just by understanding that, e.g., for an inductor,

V = L dI/dt

so that small current changes will imply very large voltage changes. If you're in to intuition, this means "voltage leads current" (i.e., the sinusoid of voltage is 90 degrees ahead of the sinusoid of current).

The above equation is also why lights in your house dim when you turn on a motor (= inductor). There is a significant current change, and the voltage changes even more, so the lights dim.

Or, you can be even more intuitive, and note that it's obvious that when you put a coil of wire (= motor = inductor) across the two sides of your house's electrical wiring, the coil of wire will pull the + and - conductors together in voltage at first. Until the magnetic field sets up to provide reactance, the coil of wire is basically a short circuit.

And finally, a capacitor is just the dual of the inductor.

<comment><USER6> "The above equation is also why lights in your house dim when you turn on a motor (= inductor). There is a significant current change, and the voltage changes even more, so the lights dim."

Uh, no. Most of the voltage drop in the source occurs in the resistance of the wiring, and you didn't even figure resistance into the equation. Intuitively, the surge current more closely matches that of a capacitor charging through resistances. It isn't even a matter of the motor inductance or the frequency of the power. In fact the early surge would be seen even if we used D.C. power and a D.C. motor. The time constant matching the light dimming involves the mass of the rotor being brought up to speed. The load is actually more resistive at startup than when an A.C. motor is up to speed and coasting.

A pure inductor starts at zero current the instant voltage is applied and it ramps up at a rate determined by the inductance, at D.C. or very low frequencies rising to where it becomes limited by the series resistance. It's pure capacitors that draw a maximum current at the instant voltage is applied to the circuit.

I believe that many people get lost in the math and lose a feel for what is happening. What you said is completely backwards from reality. Mechanical analogies are far easier for people to get a feel for than Laplace Transforms. Compare the behavior of an RLC circuit to a machanical system with a shock absorber, spring, and a mass.

People may get confused over a critically damped circuit, but the idea of a car oscillating up and down after a bump with no shock absorber is easy to grasp.

<comment><USER5> There is no way a motor is well-modeled by a capacitor, either analytically or intuitively. It's a coil of wire, practically a poster child for an inductor.

Here's a motor's equivalent circuit from Wiki:

[http://en.wikipedia.org/wiki/Induction_motor#Equivalent_circ...](http://en.wikipedia.org/wiki/Induction_motor#Equivalent_circuit)

This does have lumped resistance elements, but by the time you put the two inductors and two resistors in to the differential equation with forcing

      sin (2*pi*60*t) u(t),

you're not going to be in a good place as far as intuition goes.

In fact, overall, I'm not seeing much intuitive sense in your summary. Once you really delve in to it, the subject of starting electric motors (AC and DC) is very complex.

</comment></comment></comment></comment></comment></comment></comment><comment><USER3> The only place where I've encountered the term "impedance" is in relation to the hi-z and lo-z instrument cables that you use to connect eg. electric guitars to amplifiers. You'd sometimes see these hi-z to lo-z converters. Is this the same thing?

The video's explanation would lead me to believe that impedance only applies to AC circuits. But instrument cables don't run AC, do they?

<comment><USER2> In the case you're talking about, impedance means "characteristic impedance", i.e., one parameter which (along with speed of light in the given medium and length) defines the behavior of a transmission line. [1]

It is the same thing in the sense that mathematically both are a complex (i.e., Real plus Imaginary) relationship between voltage and current. However, a transmission line is a two-port network while a capacitor is a one-port network, so these two senses of impedance correspond to different physical phenomena. In a capacitor, the impedance defines the relationship between voltage and current at the port (i.e., between the two terminals of the cap; this is sometimes called the driving point impedance). For a transmission line, the characteristic impedance defines, among other things, the equivalent impedance seen at one end of the line when the line is either infinitely long or an integral number of wavelengths long and terminated at the other end by an impedance of the same value as the characteristic impedance.

Instrument cables do indeed carry AC signals, inasmuch as the signal on the line corresponds to the frequency of the note(s) you're playing. AC is just "non-zero frequency"---any signal which is not a constant, unchanging (DC) value is considered AC. So, 50/60Hz line power is an AC signal, the 2.4GHz signals used by your 802.11 network card are AC, and middle C (261.6 Hz) is AC.

[1] <http://en.wikipedia.org/wiki/Characteristic_impedance>

<comment><USER7> At audio frequencies, characteristic impedance is almost always irrelevant.

hi-Z is a crystal mic, guitar pickup or crystal phono pickup source.lo-Z is a dynamic or condenser mic source. Obviously, a preamp may be used as an impedance converter to lower hi-Z to loZ. There's such a preamp in the condenser mic.

The cables are hiZ or loZ only in that they're generally used with one source or the other. But that's by convention. loZ is usually a balanced XLR connection; hiZ is generally an unbalanced 1/4" connection.

hiZ is generally at around 1 megohm. loZ is archetypally 600 ohm. Line is neither; it's 10,000 to 50 or 100k ohm.

</comment></comment></comment><comment><USER8> I don't know about the rules in America, but this circuit design wouldn't be approved for use here in Europe. Circuits are not allowed to draw what is called 'reactive power', because it stresses the grid. Therefore you have to compensate it (in the simplest case using an inductor or a capacitor).

</comment><comment><AUTHOR> Part 2:

[http://benkrasnow.blogspot.com/2011/06/tutorial- electrical-i...](http://benkrasnow.blogspot.com/2011/06/tutorial-electrical- impedance-made-easy_19.html)</comment>