Remembering Roger Boisjoly: He Tried to Stop Shuttle Challenger Launch - <AUTHOR> https://www.npr.org/sections/thetwo-way/2012/02/06/146490064/remembering-roger-boisjoly-he-tried-to-stop-shuttle-challenger-launch <comment><USER2> In many ways, this is an update of the Titanic story of technological hubris, with the twist that the man who was right, who fought hard to prevent what he foresaw, who was briefly relieved when his dire predictions seemed to be wrong, was still dogged by the event for the rest of his life.

We don't know how those who were on the wrong side of the issue coped, and they should not be pressured to make public anything beyond what the inquiry required, but it seems plausible to me that those who could persuade themselves that they made the right decision, given the circumstances and despite the outcome, probably fared best. That is usually the case.

Edward Tufte tried to suggest that Boisjoly could have presented his case more effectively. Tufte may have been thinking purely pedagogically, but regardless, the implied criticism was unjustified, as Boisjoly's point should have been clear to anyone familiar with the issue, and in fact it was clear to quite a few, though unfortunately not to the few who mattered, and I doubt that, for them, a different presentation would have made a difference.

We can't always be right, and we can't all be heroes, but I hope we can all avoid being the person who said to Boisjoly, when it appeared that Boisjoly's testimony might be fatal to Morton Thiokol, that he would leave his children for Boisjoly to raise if he lost his job.

<comment><USER3> What do you consider the primary technological hubris of _Titanic_?

To be clearer - the hubris of _Titanic_ wasn't technological, it was operational - it was going too fast, in an ice field, with a lack of lookouts - beyond this, it didnt reduce speed after ice was spotted.

Yes, there were issues, damage to such a small portion of the ship _(less than 1 /3)_ shouldn't have caused it to sink (I'll note, many other ships had similar or worse flaws than _Titanic_ did).

The lifeboats on the other hand (as most commonly cited) are not a real issue - _Titanic_ started launching boats about 25 min after it struck the iceberg, which is about how long a damage survey would take. It had not launched all 20 boats by the time the forward list became so great that they were unable to launch more - this doesn't even touch the multitudes below decks, many of whom didnt speak english, or were unable to be moved up to the boat deck in a timely enough fashion.

I'd also like to share something I dug up from the internet[1]:

"As far as I can determine, Titanic is the single example of a passenger ship that sank with decks level. It was probably the only time in history when lifeboats could be launched from both sides simultaneously. The Andrea Doria sinking is a more typical event. In that case, half of the lifeboats were rendered useless by the cant of the decks. Given the testimony that Titanic was "lolling" as it foundered (listing from side to side), it is highly probable that carrying sufficient lifeboats would have raised the center of gravity sufficiently to have caused a permanent list early in the evening. In other words, the weight of those extra boats might have created a situation in which they could not have been used, anyway.

And, it is conveniently overlooked that from the moment when launching boats became necessary until the moment when it became impossible was not long enough to launch the 24 boats the ship carried. If it had carried more boats, they could not have been filled and launched properly. This is not to say that some people might not have used the un-launched boats to survive--just to point out that more lifeboats was not the answer to saving everyone aboard Titanic.

The only way to save everyone on any passenger ship is to not let it sink. That's the key, not lifeboats.

White Star and the British Board of Trade had an embarassing situation on their hands. They had lost the world's largest ship...not to mention some 1,500 irreplaceable souls. They needed an issue to divert public attention from the real problems behind Titanic's foundering. For instance the lack of lookout, the speed, and the fact that the ship was already well into the ice when it struck. Why wasn't the conduct of the voyage changed at 10:00 p.m. instead of after the accident? Or, what was wrong with the design of the ship that allowed damage to the bow to ultimately claim the whole ship? These issues go to the heart of the matter, but public attention was easily diverted by the lifeboat issue. It was a diversion that worked so well that the lack of lifeboats has become almost the only safety issue ever discussed."

[1]: [https://www.encyclopedia- titanica.org/community/threads/boar...](https://www.encyclopedia- titanica.org/community/threads/board-of-trade-and-lifeboats.1799/)

<comment><USER4> They also likely would not want to draw attention to this: [https://www.snopes.com/news/2017/01/06/coal-fire-sink-the- ti...](https://www.snopes.com/news/2017/01/06/coal-fire-sink-the-titanic/)

<comment><USER5> Why? Bunker fires were common on coal fired ships, as was dealing with them by shovelling coal into the furnace, or venting steam. It's an interesting snippet, but doesn't seem very compelling that it was a cause. I didn't watch the actual documentary, so only have the piece you linked to go on.

<comment><USER6> high speed in an ice patch -> disaster

burning more coal than necessary -> high speed

coal fire -> burning more coal

<comment><USER7> Even on a ship like the Titanic, it's not like the only way to slow down is to stop burning coal. They could have vented steam instead of using it for propulsion - that is always an option.

<comment><USER8> From a documentary on channel 4 about this, the risk was more to run out of coal before arriving as the fire would have been ongoing since the departure and been quite big.

<comment><USER9> It's not just that. It made the steel brittle. Not a good combo in freezing water.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> My main unanswered question here (I admit I haven't looked particularly hard for the answer) is what the false positive rate was. How many times have similarly major concerns raised, about things that were not ultimately an issue? Where the NASA administrators bombarded with such concerns for every launch, or were reports a relatively rare occurrence?

Designing processes to appropriately address such concerns seems to hinge on the answer.

[edit for grammar]

<comment><USER11> When the engineers are saying no, and the administrators are saying yes, you don't need to know much more.

<comment> GuB-42 Engineers will always say no. It's the administrator's job to decide which kind of "no" is acceptable.

Here, the problem is not the engineers saying no. It is that for some reason, administrators weren't able to properly discern trivial matters from life threatening ones.

<comment><USER12> >>administrators weren't able to properly discern trivial matters from life threatening ones

That suggests a very serious flaw in the overall capability, seriousness and knowledge of the project of the person making that decision.

The administrators are not that stupid. Reading the article carefully. These people were worried about the launch to be delayed by an year. That subsequently brings up the question about how that would work for their careers.

Also what is the downside for these people. Say the Shuttle crashes, they are not going to be billed $2 billion or charged with manslaughter. Heck, all they get is a job transfer in the worst possible case([https://en.wikipedia.org/wiki/Linda_Ham#Columbia_disaster_an...](https://en.wikipedia.org/wiki/Linda_Ham#Columbia_disaster_and_investigation_report)).

If you have literally no repercussions for making wrong decisions. You have now incentivized the administrators to make whatever decisions works best for them personally.

No amount and quality of engineering reports are going to work from here.

</comment></comment></comment></comment><comment><USER13> Unfortunately the management culture at NASA, something Richard Feynman also criticized, did not change. This lead to managers ignoring engineers' warnings about the foam strikes on Columbia, and also rejecting requests for high resolution images.

Linda Ham, the manager who rejected these requests left the space shuttle program after the Columbia disaster and was moved to other positions at NASA. [https://en.wikipedia.org/wiki/Linda_Ham#Columbia_disaster_an...](https://en.wikipedia.org/wiki/Linda_Ham#Columbia_disaster_and_investigation_report)

<comment><USER14> "and also rejecting requests for high resolution images."

That's a decision I don't understand. Would taking the pictures have been a big change in flight plan or something like that?

<comment><USER15> Military stuff.

Secondly, there was a strong prevalence of the idea that, even IF they had found something, nothing could have really been done, which was probably true according to the post-mortem.

<comment><USER14> "even IF they had found something, nothing could have really been done, which was probably true according to the post-mortem."

That would be a pretty weak excuse and only applicable in hindsight. I have read that depending on the size of the damage they could potentially have changed the descent profile to put more load on the other wing. To decide this you first need the data.

<comment><USER16> A fascinating article was posted a few years ago outlining the possibility of scrambling the _Atlantis_ orbiter while _Columbia_ was in orbit and performing a rescue of her crew. It was a fascinating "what-if", well worth your time:

[https://arstechnica.com/science/2016/02/the-audacious- rescue...](https://arstechnica.com/science/2016/02/the-audacious-rescue-plan- that-might-have-saved-space-shuttle-columbia/)

</comment></comment></comment></comment></comment><comment><USER17> There are commenters here who in usual HN form speak as if they would have acted differently than the team at NASA did if they had been in involved. The reality is that it is unlikely.

Business schools use variations of the Challenger launch as a case study in group decision making and organizational behavior. I experienced one while at business school. The crucial parts of Challenger were applied to another scenario. Risk and safety were brought up in discussion but were outlier considerations by the group. The group agreed to proceed with the plan.

<comment><USER18> I took that case study in business school at Cornell. I was one of the 4 people in the class that day (there must have been 30-40 folks in the room) whose recommendation was to NOT launch. I felt so pleased when the prof revealed that the case was a disguised version of the Challenger incident.

EDIT: In response to the suggestion, below I've removed the reference to actual case study, to allow future students to enjoy it as much as I did.

<comment><USER17> Yep. That's the one. The author of the case was my professor. You might want to omit the details of the case in your comment because folks here might experience it one day. :)

I was the _only_ dissenter in my class. You and I should connect.

<comment><USER18> We should definitely connect! :-) Removed the reference to the actual case, as you suggested.

<comment><USER19> Aw, come on. I'm never going to go to business school. Let me spoil it for myself if I so choose... Just use a URL shortener so the link doesn't give away information for those who don't want it spoiled.

<comment><USER20> I don't know why you are getting downvotes, I'm not going to be heading to business school and wouldn't mind reading it either.

</comment></comment></comment></comment></comment></comment><comment><USER11> Not widely known is that it was Sally Ride who clued in Feynman ['s wingman], secretly. If it had got out, she would never have flown again.

Also not widely known was that she was gay. She had to conceal that, too, from NASA. She died young of pancreatic cancer.

<comment><USER21> The story people read in Feynman's autobiography was that General Donald Kutyna came by and made some very knowledgeable suggestions about areas worthy of investigation. Kutyna revealed after her death that Sally Ride clued _him_ in, and he in turn passed the knowledge on to Feynman.

 _Kutyna: On STS-51C, which flew a year before, it was 53 degrees [at launch, then the coldest temperature recorded during a shuttle launch] and they completely burned through the first O-ring and charred the second one. One day [early in the investigation] Sally Ride and I were walking together. She was on my right side and was looking straight ahead. She opened up her notebook and with her left hand, still looking straight ahead, gave me a piece of paper. Didn 't say a single word. I look at the piece of paper. It's a NASA document. It's got two columns on it. The first column is temperature, the second column is resiliency of O-rings as a function of temperature. It shows that they get stiff when it gets cold. Sally and I were really good buddies. She figured she could trust me to give me that piece of paper and not implicate her or the people at NASA who gave it to her, because they could all get fired._

 _Kutyna: I wondered how I could introduce this information Sally had given me. So I had Feynman at my house for dinner. I have a 1973 Opel GT, a really cute car. We went out to the garage, and I 'm bragging about the car, but he could care less about cars. I had taken the carburetor out. And Feynman said, "What's this?" And I said, "Oh, just a carburetor. I'm cleaning it." Then I said, "Professor, these carburetors have O-rings in them. And when it gets cold, they leak. Do you suppose that has anything to do with our situation?" He did not say a word. We finished the night, and the next Tuesday, at the first public meeting, is when he did his O-ring demonstration.

We were sitting in three rows, and there was a section of the shuttle joint, about an inch across, that showed the tang and clevis [the two parts of the joint meant to be sealed by the O-ring]. We passed this section around from person to person. It hit our row and I gave it to Feynman, expecting him to pass it on. But he put it down. He pulled out pliers and a screwdriver and pulled out the section of O-ring from this joint. He put a C-clamp on it and put it in his glass of ice water. So now I know what he's going to do. It sat there for a while, and now the discussion had moved on from technical stuff into financial things. I saw Feynman's arm going out to press the button on his microphone. I grabbed his arm and said, "Not now." Pretty soon his arm started going out again, and I said, "Not now!" We got to a point where it was starting to get technical again, and I said, "Now." He pushed the button and started the demonstration. He took the C-clamp off and showed the thing does not bounce back when it's cold. And he said the now-famous words, "I believe that has some significance for our problem."_

[https://www.popularmechanics.com/space/a18616/an-oral- histor...](https://www.popularmechanics.com/space/a18616/an-oral-history-of- the-space-shuttle-challenger-disaster/)

<comment><USER11> It is a deep indictment of NASA that any of that skulduggery was necessary. I don't doubt that it was. My guess at the reason is that allowing people to talk about things that were true would have led to the whole program being canceled.

The program was such a disaster, by every conceivable reckoning; using the money wasted on that white elephant, we could have had, thirty years ago, what Elon Musk is just finally getting around to, and (as a bonus!) without Elon.

<comment><USER22> Why would it be better without Elon?

<comment><USER23> Perhaps he means that Musk has the tendency to leave his ego in the driver's seat. One false move, one unlucky moment, and it can cause another tragedy.

Otherwise one person achieving what Musk has is just as good as any other. Just as long as they are stable enough to not pose a risk to the mission.

</comment></comment></comment></comment></comment><comment><USER24> It is much easier to prevent disasters like this if you design processes to surface problems as early in the process as possible. It does not take anything away from Roger Boisjoly to say that it's really hard to stop the process once you get far enough toward launch/release/production or whatever the end result is. The real failing of the NASA managers is that they failed to create a culture of safety that would encourage problems to be addressed long before they became catastrophes.

<comment><USER25> This is easier to do if it's anonymous, in general. One I've seen is hand out decks of cards, have people put in a card, and if there isn't consensus on moving forward, things stop.

[https://retromat.org/en/?id=130](https://retromat.org/en/?id=130) is an example of this. Trust is the most important thing about teamwork imo.

<comment><USER26> Anonymous voting is only necessary because people don't trust each other.

If you trust each other there is no need for anonymity.

<comment><USER27> Yes but the inverse isn't true either: removing anonymity doesn't cause everyone to trust each other. If you don't have trust, or you're not sure, an exercise like this is a good idea

</comment></comment></comment></comment><comment><USER28> Also worth remembering Bob Ebeling, another Thiokol engineer who was wracked with guilt for 30 years, thinking he should've done more to stop the launch. [https://www.npr.org/sections/thetwo- way/2016/02/25/466555217...](https://www.npr.org/sections/thetwo- way/2016/02/25/466555217/your-letters-helped-challenger-shuttle-engineer- shed-30-years-of-guilt)

</comment><comment><USER29> This got me following the rabbit hole and I found this NYTimes article from 1987 about the subsequent lawsuit against Morton Thiokol by Boisjoly.

[https://www.nytimes.com/1987/01/29/us/engineer-who- opposed-l...](https://www.nytimes.com/1987/01/29/us/engineer-who-opposed- launching-challenger-sues-thiokol-for-1-billion.html)

If you forgot how different things were back then, here are a few highlights:

Thiokol spokesmen ''consistently and falsely'' portrayed Mr. Boisjoly as ''a disgruntled or malcontented employee whose views should be discounted and whose professional expertise should be doubted,'' the suit said. It cited press interviews in which Thiokol spokesmen labeled Mr. Boisjoly a ''tattletale'' and an ''impatient'' employee who tried to hire subcontractors in violation of a contract.

Roger stood his ground and paid dearly for it, kudos to him for having some integrity.

</comment><comment><USER30> Edward Tufte made a great case that data visualization could have prevented the launch... and that PowerPoint was to blame.

[https://www.asktog.com/books/challengerExerpt.html](https://www.asktog.com/books/challengerExerpt.html)

<comment><USER21> Roger Boisjoly, and others, have a response to Tufte's analysis that is worth reading:

[https://people.rit.edu/wlrgsh/FINRobison.pdf](https://people.rit.edu/wlrgsh/FINRobison.pdf)

Tufte made a number of errors, including the sorts of errors Tufte might freak out about:

 _The other temperatures Tufte lists on Table 1 are of the ambient air at time of launch. Tufte has mixed apples and oranges. Tufte thus has both coordinates on the scatterplot wrong. The vertical axis should be "blow-by", not "O-ring damage", and the horizontal axis should be "O-ring temperature", not a mixture of O-ring temperature and ambient air temperature._

I don't know if he ever offered a response to this paper.

<comment><USER30> Tufte's improvement on the NASA chart was _adding_ an X/Y axis to show correlation between temperature and damage; something the original charts failed to communicate.

Maybe he didn't offer a response to this paper because the fundamental design approach is correct, albeit the actual numbers need refinement?

<comment><USER31> _> Tufte's improvement on the NASA chart was adding an X/Y axis to show correlation between temperature and damage; something the original charts failed to communicate._

But, as the response that justin66 linked to shows, Tufte's "improved" chart is incorrect on both axes: its "temperature" axis wrongly conflates ambient air temperature with O-ring temperature, and its "damage" axis wrongly conflates erosion with blow-by. (See comment at end.)

The reasons why the charts shown by the Thikol engineers did not communicate "correlation between temperature and damage" was, as the Boisjoly response makes clear, because, at the time (the night before the Challenger launch), _nobody knew what that correlation was_. The argument the engineers were making was _not_ "the risk of O-ring failure increases with decreasing temperature". It was "since we don't understand the root cause of the O-ring issue, we should not launch at any temperature outside the previous range of launch temperatures". The lowest previous launch temperature was 53 F; the temperature on the morning of the Challenger launch was 29 F.

Why did the engineers not know the correlation between temperature and damage? Because the data they had at the time was inconclusive and incomplete (for example, they did not even have complete data on the ambient air temperature and the O-ring temperature for every previous launch--a point Tufte overlooks), and their attempts to obtain more data had been mostly unsuccessful.

And why were the engineers reduced to making what is, on the surface, a fairly weak argument the night before the Challenger launch? Because, as I noted in another comment upthread, they had already tried, the previous summer, to get NASA to stop _all_ Shuttle flights until the O-ring issue could be properly understood and fixed, on the grounds that with it not fixed, _every_ Shuttle flight had a significant risk of loss of vehicle and loss of life (see further comment below on this). And NASA _refused_. So the engineers that night already knew they were dealing with a NASA management that was simply ignoring a critical flight risk; therefore, arguments of the form "this is a critical flight risk we don't understand, so we shouldn't launch" were out of bounds, since they had already been tried and had failed. The engineers were simply trying to do the best they could to get at least _some_ Shuttle flights stopped, and making the best arguments they could to do that, against the background of their much better argument for having _all_ flights stopped having already failed.

A further comment on what I said above, that _every_ Shuttle flight was a significant risk with the O-ring issue not understood. Tufte's assumption that the root issue was in fact a "correlation between temperature and damage" was _wrong_. It is true that the data from previous launches showed more "witness events" (evidence of either erosion or blow-by) at lower temperatures. But the engineers also had test stand data showing that under some conditions, the O-ring joints were failing to seal at any temperature below 100 F! (And there was at least one flight that showed blow-by, i.e., evidence of the O-ring joint failing to seal, at 75 F.) So the problem wasn't "the joint is OK at higher temperatures but unacceptably risky at lower temperatures". The problem was "the joint is unacceptably risky at any temperature below 100 F"; the fact that it was more unacceptably risky at lower temperature than higher was a relatively minor detail. But NASA had already refused to listen to that argument.

And a final brief comment on erosion vs. blow-by. Erosion is damage to the O-ring due to hot gas eroding part of it while the O-ring is sealing the joint. Blow-by is hot gas going right past the O-ring because it is _not_ sealing the joint. The O-rings were designed to tolerate a certain amount of erosion, based on the expected temperature of the gas and the duration of the burn; so erosion, in and of itself, was not evidence of a problem not anticipated by the design. But the O-rings were _not_ designed to not seal at all: not sealing was a failure of the design. So blow-by was direct evidence of a failure of the design. That's why conflating the two is wrong: blow-by is the problem, not erosion. (It's true that an O-ring that has hot gas blowing by it because it's not sealing will also have erosion; but blow-by without erosion is still a problem, while erosion without blow-by is not. So blow-by is the indicator that should be focused on when assessing the risk level of the O-ring joint design.)

<comment><USER30> #ChangeMyView delta awarded :-)

</comment></comment></comment></comment></comment><comment> mark-r Needs a (2012) appended to the title. A very powerful story deserving to be retold.

</comment><comment><USER32> > "We were talking to the people who had the power to stop that launch."

The problem is that the people who had the power to stop the launch were managers dealing with other concerns. Engineers should have the power to stop the launch.

And with principles like Agile and Lean, engineers are fortunately increasingly getting empowered to stop a launch if they feel it would be irresponsible. I hope NASA now uses these sort of principles too.

<comment><USER33> I have to disagree that Agile is any kind of solution to this. I've seen it fail too many times. If implemented correctly, maybe, but that rarely happens. And why is that? Why so many Agile horror stories?

There is a real lack of leadership in the world, at all levels of society. It's not something an ideal like agile can fix. Without good leaders we end up with broken agile. I'm pretty sure NASA has a good engineering process. It doesn't matter if nobody listens.

<comment><USER32> Agile is not a thing that fixes things. Agile is a set of principles. People over process, and that sort of thing. Blindly implementing Agile as a process ignores that basic principle of Agile.

In this particular case, there was a person, and expert, who saw a very real danger and was overruled by people who lacked his expertise because his opinion was inconvenient to them. That's not good. If an expert sounds the alarm, you listen to him, no matter how inconvenient it may be.

And in Agile and maybe more explicitly in Lean, the people who do the work are empowered to make decisions regarding their work. Responsibility shouldn't lie entirely with people who may be distracted by other concerns.

</comment></comment></comment><comment><USER34> Serious question: What was the proposed course of action had they decided to cancel the launch? My (possibly wrong) understanding was that the O-ring damage only would have been apparent if they disassembled one or both of the solid rocket boosters. Was anyone seriously proposing to do this prior to the launch?

The question is relevant because I've read conflicting sources about whether the damage to the O-ring caused by the freezing temperature was permanent. If the temperature permanently compromised the O-ring material, then a delay wouldn't have saved the Challenger, only a disassembly of the boosters would have. However, if the O-ring performance would have recovered when brought back to a normal temperature range, then a delay could have prevented the disaster. Does anyone have any definitive sources on this?

<comment><USER35> The O-rings were not damaged prior to launch, but they didn't function if they were too cold. You'd have to wait until a warmer day.

</comment></comment><comment><USER36> The morning before the shuttle blew up I was watching the news on tv before catching the bus to school. You could tell just looking at the news feed that that the shuttle was covered in ice. Even to my 12 year old self it seemed insane to launch a space ship into orbit when covered in ice. I was so confused when told in school that the shuttle blew up. I was like, no way would anyone think that was a good idea to launch today. I still to this day don't know if they de-iced it first or just shot it up there and hoped for the best.

</comment><comment><USER37> I wonder how many other times an engineer advised NASA to delay a flight.

</comment><comment><USER38> If I remember right the roberts report on this said each time they launched at a low temperature any degradation of the seal became the new normal and "safe". The launch temperature got lower and lower and each time the normal was reset.

</comment><comment><USER39> Gotta get better at ringing the right alarms and persisting if you want to be heard. He could've done lots more things to warn but he didn't: tried to warn astronauts' families, leak it to the media, and such. But he didn't. Why lionize him for failing?

</comment><comment><USER40> This story should be shown to every manager who ignores engineers recommendations.

</comment><comment><USER41> Pretty sure an actor played him in the movie about the challenger disaster.

[https://imdb.com/title/tt2421662/](https://imdb.com/title/tt2421662/)

</comment><comment><USER42> off topic: if I decline the cookie policy, I get the plain text site! That's an extra free benefit. Love it.

</comment><comment><USER43> GDPR-compliant link: [https://text.npr.org/s.php?sId=146490064](https://text.npr.org/s.php?sId=146490064)

<comment><USER44> Oh wow. Better presentation too.</comment></comment>


Stack News – stay up-to-date with the tools you care about - <AUTHOR> https://stackshare.io/news?ref=hackernews <comment><USER2> Cross-checked the top posts for React and Elm in the last 24 hours on HN and /r/programming...8 for 8. :applause: ok, this could make my morning click- around way quicker. Good find!</comment>


Experiences of building a new ERP system in (GemStone) Smalltalk - <AUTHOR> http://smalltalk-bob.blogspot.com/2016/09/a-year-goes-by.html <comment><AUTHOR> The previous post has more technical details: [http://smalltalk- bob.blogspot.com.au/2016/01/lessons-learned...](http://smalltalk- bob.blogspot.com.au/2016/01/lessons-learned.html)</comment>


Startup idea:  help people avoid buying books (without infringing copyright). - <AUTHOR>

Given a book as the query, the service would return a collection of web links to comparable free content on the web.
I guess the main issue is this:  what motivation can one use to encourage people to maintain such book-centered link collections for the benefit of others?
Perhaps an advertisement-based profit-sharing system might work? <comment><USER2> It is called a search engine. Cuts out the need for manual submissions and skirts copyright issues by claiming to offer an unrelated service (indexing).

<comment><AUTHOR> Yes, but there's effort involved in using it. A student would appreciate having a collection of links available as an alternative to a textbook say.

</comment></comment><comment><USER3> Sounds like you're describing curated search for a particular set of queries (text books). Isn't this a subset of what Mahalo does?</comment>


Paul Krugman: The austerity delusion - <AUTHOR> http://www.theguardian.com/business/ng-interactive/2015/apr/29/the-austerity-delusion <comment><USER2> What Krugman and his ilk fail to address is capital misallocation. Placing capital into its most productive use is the definition of a well functioning market.

We tolerate market distortion for greater social good: safety regulations, environmental regulation, labor laws, etc.

Unfortunately, with QE1,2,twist,3 and ZIRP, we are now well past properly functioning capital markets.

At a gross scale, every single investment in the world now is actually a bet on central bankers keeping interest rates suppressed, liquidity flowing, and money printers running.

The trillions of dollars of distortion that capital markets have experienced since 2008 has fundamentally altered the investment landscape. And not for the better.

Krugman crowing about austerity is laughable. Of course, government spending is just another form of capital distortion. Sure, try some more. It'll be another drop in the bucket. It will not move the needle in a sustainable fashion.

Global demand is being vacuumed up by economies through competitive currency devaluation. Deficit spending (Krugman's argument) generates new demand in the hopes of creating jobs and nurturing the economy.

Unfortunately, since 2008, our global financial system is so radically distorted that a burst of demand isn't about to lift anybody out of long-term economic doldrums.

At the root of the problem is massive capital misallocation that has been baked in the cake. Global supply is so large due to QE and ZIRP that economies worldwide cannot support their investments.

Instead of market forces washing out the misallocated capital, we've entered into a global extend-and-pretend scheme. How and when this ends is anyone's guess.

Krugman doesn't get it.

<comment><USER3> > Instead of market forces washing out the misallocated capital, we've entered > into a global extend-and-pretend scheme. How and when this ends is anyone's > guess.

So while Krugman offers data and concrete, falsifiable predictions that generally prove correct - for example, that hyperinflation will not be a problem, that QE won't move the economic needle much (but is better than nothing), that interest rates will remain low for the foreseeable future despite claims to the contrary, you offer vague handwaving about capital misallocation and "distortions" and then tell us you can't answer how and when Krugman will actually be wrong.

Can you understand why people have difficulty coming to terms with your beliefs? To me it just seems rooted in some cognitive bias about suffering and sacrifice; it sounds like you just can't believe things are this easy, that we _have_ to suffer and be punished for our vile profligacy.

<comment><USER2> First, the market is made up of all of us seeing similar data and coming to our own conclusions based on our interpretations and self-interest.

Second, the market didn't just crash in 2008, it melted down. There were fears of simultaneous runs on global banks with catastrophic consequences. Bernanke/Geithner/Paulson orchestrated government backstops on money market funds, on Fannie/Freddie debt, on shotgun-marriages between banks, on defusing the credit default swap nightmare constructed by AIG and its counter parties. ZIRP and QE were brought in to juice spirits in the hopes of rescuing growth.

2008-2009 was the end of the financial world as we knew it going back to Paul Volcker. It was nightmare. It was a catastrophic end that many and been predicting for some time. Some saw the writing on the wall by 2005 and were positioning their investments accordingly. Read about the thoughts of Ray Dalio, Stan Druckenmiller, or Jeremy Grantham.

Since 2008-2009, we're now in a whole new regime. Perpetual QE + ZIRP is not in ur economic textbooks and I do not believe our models can tell us what is coming.

We had an opportunity when things were stable in 2010 to find a sustainable path forward. Instead, worldwide, we've just taken regular hits of QE.

<comment><USER3> > Perpetual QE + ZIRP is not in ur economic textbooks and I do not believe our > models can tell us what is coming.

That very situation is what Krugman did a lot of his academic work on. So far it's played out mostly like what he and other mainstream economists predicted. You can always say "until it doesn't", but that's the nature of life. We have to go with what we know.

It's not as if Krugman _wants_ QE + ZIRP; he'd rather get out of it ASAP with his _actual_ policy prescription (fiscal policy), but he's had to settle for QE/ZIRP.

> Yet there are many economists, myself included, who regard this view [that > QE will be effective and give us a strong recovery] as highly unrealistic, > yet support more aggressive Fed action all the same. Why? First, because it > might help and is unlikely to do harm. Second, because the alternative — > fiscal policy — may be of proven effectiveness, but is also completely > blocked by politics. So the Fed's efforts are all we have.

[http://krugman.blogs.nytimes.com/2013/07/27/miltons- paradise...](http://krugman.blogs.nytimes.com/2013/07/27/miltons-paradise- still-lost/)

<comment><USER2> > So far it's played out mostly like what he and other mainstream economists > predicted.

Oh christ on a cracker.

Mainstream economists should have all taken their PhD diplomas off their walls and collectively shredded them in the fall of 2008.

Under the watchful eyes of mainstream economists at the Federal Reserve, the watchful eyes of academics at institutions worldwide, the global financial system melted down.

Now, we are to believe that these same folks who couldn't see the 2008-2009 crisis know exactly how to lead us out.

</comment></comment></comment></comment></comment><comment><USER4> If all you care about is the short-term, then forget about austerity. Juice things up a bit. Throw some borrowed money at it. Alleviate some economic misery. But long-term, living within your means leads to greater stability, a market where prices reflect reality, and people that save are rewarded. This is a far greater payoff in exchange for some potential short-term pain.

The worst part is that many governments claiming "austerity" are just playing accounting tricks and not really moving towards solvency. The solution for having spent too much and distoring the economy is not to continue to do the same.

We'll never know who is right, though. Most of the world has been following the recommendations of this "economist" for decades. Maybe only indirectly because I'm not sure how many people pay attention to him. Spending massively is popular with the public because they get all the benefits, but will be 6 feet under when the bill comes due.

Just because it's a government instead of a household or a business does not making borrowing against your future when you have no capability to pay it back a good idea. This guy is the same type of person that thinks that companies carrying a lot of debt is a good idea. The sign of a healthy business. Bunk.

<comment><USER5> The proper time for austerity is when things are good. When things are bad, juice it up, especially if you can borrow at rates that are negative when inflation adjusted.

The problem with austerity now to make the long term better is that it's not a zero sum game. Austerity lowers GDP growth, which can offset any move toward solvency. You spend less and surprise, you also take in less. Eventually you recover, but poorer than if you had juiced things a bit and no less in debt.

<comment><USER6> Had the government in the UK run a surplus prior to the financial crisis I suspect the case for a larger Keynesian response from the UK public would have been better received. That is afterall what Keynes originally proposed. Modern Keynesians have disproved it's necessity and yet... public opinion ultimately drives policy, perhaps Keynes requirement of running a surplus in the good times in order to run a deficit in the bad was more about satisfying the psychology of the general public than anything else.

<comment><USER7> The UK will have a big problem in 10 years when there are still no jobs for young adults.

</comment></comment></comment></comment><comment><USER8> What about Japan? They tried to buy economic growth (dubbed Abenomics) and failed horribly. The economy is worse and government debt is higher than ever.

Models that can't explain some weird edge cases are ok. But what happened in Japan is the exact opposite of what Mr. Krugman tries to convince people.

<comment><USER3> Krugman has written a lot about Japan, and Abenomics. According to Krugman's own analysis, Abenomics would not be very effectual and warned that particular Abenomics policies (like the tax increases) would indeed worsen the economy. It's a little disingenuous to say the economy is worse: growth did improve (until the VAT increase Krugman warned would be disastrous) and unemployment did fall, and Krugman wasn't worried about the debt in the first place since the Japanese can still borrow at very low rates.

Abenomics does not correspond almost at all to Krugman's prescriptions, it was simply expected to be slightly less bad than what was happening before.

<comment><USER9> To get that growth, Japan massively debased the Yen, chopping down the Japanese standard of living and reduced the real value of all of that debt Japanese citizens are expecting to be paid back by the government.

If you have to significantly harm your currency to get growth, what you're getting is not actual growth, merely an illusion.

<comment><USER3> In what sense do you mean Japan has "debased" the yen? Inflation has been virtually nonexistent. A weaker yen promotes exports; obviously not something that is automatically bad for the economy. It's absolutely not true that having a weaker currency means growth is illusory.

<comment><USER9> If you reduce the value of your currency by 25%, and get 3% GDP growth in the process, you have to adjust the baseline lower and what you've actually got is a drastic reduction in real economic value.

The Japanese government has been aggressively devaluing the Yen, because they're bankrupt and can no longer afford to pay even the interest on their debt without either using printing or new debt issuance. They're stuck in a downward spiral. Japanese savings have collapsed to zero, so the Japanese people can no longer afford to fund continued government debt expansion as they were before. That has left the Japanese government with no other options but currency debasement, ie devaluing the real worth of outstanding Yen based debt (which is mostly held by Japanese citizens, which is then ultimately a slashing of their wealth).

Japan has suffered zero deflation over the last 25 years. That's why Japan is one of the most expensive places on earth to live in most every respect. If they had suffered 25 years of deflation, their prices would have gone down accordingly.

Their long-term price index shows zero deflation.

[http://i.imgur.com/Cvwj26A.png](http://i.imgur.com/Cvwj26A.png)

The notion they've been stuck in deflation is keynesian economists confusing what deflation is. Under keynesian economics, if a bubble is inflated, and then bursts, taking down real estate prices, they call those falling prices deflation - when in fact it is not. This is one of the many obvious flaws at the heart of keynesian economics, which has helped lead the globe into its current perpetual stimulus addiction path.

<comment><USER10> > If you reduce the value of your currency by 25%

It is the dollar that has appreciated, not the yen that has depreciated. The dollar has appreciated against most currencies in the last year – probably because people expect interest rate hikes this year. It has nothing to do with Japan per se.

</comment></comment></comment></comment></comment></comment><comment><USER11> Interest rates are 0% and our economy isn't growing (Q1 '15). I find that scary.

I'm seeing the results of Krugman's philosophy on the ground: Asset prices are way up. Housing in my city is almost as expensive as it was before the '08 crash. For those of us that are lucky enough to own stocks and a house -- three cheers for Krugman, 0% interest rates, and QE!

But the asset economy is outpacing the real economy. Wages are not up. Working age labor force participation is down. Q1 GDP was flat. If you're a working person in the U.S. right now, you're not thriving.

And to me this is a sign that Keynesianism is perhaps played out. We're increasing asset prices, making housing more expensive and increasing inequality, without actually growing the economy.

<comment><USER12> Q1 '15 is being considered a blip for many reasons (weather, west coast port strike, the strong dollar [1], big hits in the energy markets and markets affected by energy, etc.). Many economists expect a bounce back, much like last year.

1\. Remember when people, counter to what Krugman said, said the dollar was going to experience hyperinflation?

</comment></comment><comment><USER13> the main problem with "austerity" isn't the theoretical question of the spending levels, it is the political&social issue that practical "austerity" means that during bad times poor people must pay for the well connected and rich people's exuberance of the good times. "Sorry, we have to reduce your pensions and salaries to pay back the loans we spent on our luxury BMWs."

</comment><comment><USER14> Related article: "A jobs miracle is happening in Britain, thanks to tax cuts. Why don't the Tories say so?"

[http://www.spectator.co.uk/features/9475222/the- coalitions-j...](http://www.spectator.co.uk/features/9475222/the-coalitions- jobs-record-is-miraculous-why-wont-they-talk-about-it/)

<comment><USER15> Unfortunately that article doesn't really explore how much these people who are part of the jobs miracle are being paid!

</comment></comment><comment><USER16> Paul Krugman in 2002: "To fight this recession the Fed needs…soaring household spending to offset moribund business investment. [So] Alan Greenspan needs to create a housing bubble to replace the Nasdaq bubble."

Somehow every op-ed this guy writes gets treated as the word of god. It is important to remember that while he does have a Nobel prize, he has also said a lot of profoundly dumb stuff.

I am not saying that Austerity is inherently good or inherently bad. The truth, like for everything else, lies in the grey and is situation dependent. Krugman however is absolutely religious about fiscal stimulus and demand side economics, and is going to claim that the solution to EVERY problem is increased government spending.

These situations are nuanced and unfortunately what you are hearing from Krugman is something more akin to proselytizing his religous views on the graces of Government spending(a view which no facts or events would every sway him from) opposed to thinking critically about a difficult problem.

<comment><USER17> I read the editorial that the "housing bubble" quote is from and it doesn't sound at all like he actually recommends it:

[http://www.nytimes.com/2002/08/02/opinion/dubya-s-double- dip...](http://www.nytimes.com/2002/08/02/opinion/dubya-s-double-dip.html)

The theme of that article seems to be, "we're screwed, and here's why." In that context, the quote in question seems to be saying, "the only way out would be to create a housing bubble, that's how screwed we are."

I think it's ironic to complain about "treated as the word of god" in a comments section absolutely flooded with criticism of him.

I wish we could discuss the ideas rather than the author, in any case.

<comment><USER16> These are after the fact justifications.

He does indeed say that lowering rates to inflate asset prices would be our easiest way out of the situation.

He goes back only to say that it is unlikely that we will cut rates enough so it porbably won't work (ie this will fail because our attempt to create bubbles will be too half assed to work.)

He's pretty clearly serious that lowering rates to encourage investment is our ticket out of the problem.

<comment><USER17> I'm curious, why is my version an after the fact justification, but yours isn't?

<comment><USER18> I believe that amn89 was saying that _Krugman 's_ version was an after the fact justification.

</comment></comment></comment></comment></comment><comment><USER19> The problem with austerity as I see it is that it is a losing strategy if any other player doesn't adhere to it. Example - Economy A starts QE, Economy B sticks to austerity. Economy A gets super cheap money they have to invest somewhere, Economy B is short on money due to austerity. Economy A with the money out of thin air buys most of the industrial assets of Economy B. Economy B wants to change the course by rejecting austerity and starting QE. But the first mover advantage is gone, Economy A's stocks are already inflated so Economy B can't gain anything by trying to do the same as Economy A, buying Economy A's industrial assets, causing downfall of its trading currency.

</comment><comment><USER20> Krugman presents this as a dilemma between two choices.

1\. Institute austerity now. 2\. Institute stimulus now, pay down debt responsibly once the good times roll.

This is a sham dilemma. The real options appear to be.

1\. Institute austerity now. 2\. Institute stimulus now, but half-ass the debt paydown in the future, condemning future generations to a high debt to GDP ratio.

Given this real dilemma, which is worse? I don't pretend to know the answer. I know austerity sucks. I also know that governments lack the self discipline to pay down debt significantly in the future.

Which option is actually worse?

<comment><USER3> > Given this real dilemma, which is worse? I don't pretend to know the answer. > I know austerity sucks. I also know that governments lack the self > discipline to pay down debt significantly in the future.

What real-world problems do high debt/GDP ratios cause?

Isn't the whole point of stimulus to kick-start the denominator in the debt/GDP equation?

At any rate, governments don't need to "pay down debt" (which is kind of a strange thing to say, since the government is always paying off bonds.) They just need to grow the economy such that the debt is less substantial, which pretty much everyone has always done. I mean, the US wasn't going through times of tribulation paying off its 100%+/GDP debt after WWII, we just inflated it away.

<comment><USER20> Here's an interesting meta-study about the effects of a high debt to gdp ratio.

[http://www.nber.org/papers/w16827](http://www.nber.org/papers/w16827)

I admire the candor of your second point ("we just inflated it away.") I think this is ultimately what will happen here in the USA.

I think that most people would choose austerity now over higher inflation down the road, but maybe I'm delusional. Our consumer savings rates continue to trend down so maybe people just wouldn't care much.

[http://www.tradingeconomics.com/united-states/personal- savin...](http://www.tradingeconomics.com/united-states/personal-savings)

<comment><USER3> > Here's an interesting meta-study about the effects of a high debt to gdp > ratio.

I know you're sincere, but I kind of had to laugh. Reinhart and Rogoff have been through the ringer for publishing that paper, which is almost entirely bunk. There was a huge storm that made it even to the mainstream media about how truly flawed that paper was (with errors getting as basic as "they can't use Excel.") It's even mentioned in the OP.

To be fair, the claim that slower economic growth is correlated with high debt is somewhat true. But most argue causation is in the other direction: slow growth causes high debt.

> I admire the candor of your second point ("we just inflated it away.")

You might recall that the time period in which we inflated that debt away was one of the most prosperous for all Americans.

Meanwhile, austerity will directly lead to people losing their jobs and an economic slowdown that will prolong the suffering people are already going through...for what?

<comment><USER20> I'll concede the point on the R&R paper. Didn't know there was so much controversy about it, and the accusations that some of their findings were due to a technology glitch are pretty damning.

But still, the median age of the US population in 1950 was 30 years old. Today it's over 37 and climbing. And (not to beat a dead horse) the savings rate is half what it was then.

To me that indicates that a far larger percentage of our population is going to be living on (paltry) fixed incomes than at any other time. Inflation would be crushing to our aging population.

I just don't think you can compare our situation to the 1950's.

</comment></comment></comment></comment></comment><comment><USER21> Looks like BoE Governor Carney is anti-austerity. I wonder how his meetings with Mr. Cameron go.

[http://www.theguardian.com/business/2015/jan/28/bank- england...](http://www.theguardian.com/business/2015/jan/28/bank-england- governor-attacks-eurozone-austerity)

</comment><comment><USER22> I'm sure the Greeks would be delighted to hear that more debt spending is the shot in the arm they've been looking for.

<comment><USER23> He specifically calls out Greece as a special case. His comments are directed towards economies which are not like Greece.

<comment><USER24> Ah. The trusty "No True Scottsman" fallacy...

<comment><USER3> No, it's not a "No True Scotsman" fallacy. Greece, crucially, does not print its own currency and there was no real lender of last resort available at the time.

It's been a trivial exercise for many years to point out that countries with even more debt than Greece are not going into debt crises for those exact reasons, with Japan as the almost canonical example with its still-low interest rates.

</comment></comment></comment></comment><comment><USER25> The big Keynesian idea is that somehow gov't can allocate capital better than private businesses. But that patently cannot be true. (I never could understand how Keynesian economics is different from socialism.)

<comment><USER26> Not true whatsoever.

Here's the definition as stated by Wikipedia, which I find to be pretty accurate and concise:

> Keynesian economics is the view that in the short run, especially during > recessions, economic output is strongly influenced by aggregate demand > (total spending in the economy).

This contrasts with the earlier theory that production creates demand.

And while Keynesian economics does suggest that the government should intervene in the economy, it's far from a command economy.

</comment></comment><comment><USER27> Pretty much every government could benefit immensely from austerity. The problem is, of course, that government is lousy at execution and overly political and self-preservation all and so, for example, cuts the most visible programs first.

Yes, I understand the notion of spending your way out of a recession.

</comment><comment><USER28> The _Paul Krugman delusion_ : thinking a little or a lot more of the policies that have produced little to no improvement will suddenly work if we do it _just one more time_.

<comment><USER10> Actually, The _Paul Krugman delusion_ is the illusion that people have that they have much more insight in economics and economic policy than Paul Krugman, despite never having studied economics, or read a textbook or a journal in economics.

<comment><USER29> That's really just an instance of the Dunning-Kruger effect, but it's a good example of it.</comment></comment></comment>


Postgres high-availability cluster with auto-failover and cluster recovery - <AUTHOR> https://github.com/nanopack/yoke <comment><USER2> Can we just start posting "Jepsen or it didn't happen" on these?

Clustered relational DBs are really remarkably hard to get right. A shocking percentage of projects and vendors don't really even make a serious try (cf GaleraDB.)

This project may or may not have done a reasonable job (no clue,) but the lack of information about how they tested it is a bad sign.

<comment><USER3> Most (but not all) MySQL / PostgreSQL HA systems have similar semantics of Redis Sentinel. They don't try to achieve any linearizable semantics, but to perform failure detection and fail over to some slave, with heuristics about what slave to pick. All this kind of solutions will fail to pass Jepsen tests since they are not designed to achieve the kind of strong properties Jepsen will test for. The behavior of those systems is acceptable in many production situations (it's up to the product requirements to understand if it is ok that a given set of failure modes produce data loss and/or inconsistencies) but the important bit is that the semantics is very well documented so that users know what tradeoffs they are making.

EDIT: Shameless plug: I believe it was a shame that Sentinel was initially badly received because of the Jepsen test (now tons of users are using Sentinel with success, fortunately), since for what it does, it is very advanced compared to * SQL failover solutions, so a porting of Redis Sentinel to failover those systems could make sense IMHO. A few things Sentinel has that are desirable in * SQL failover systems:

1\. It's distributed so Sentinel itself is not a single point of failure.

2\. When the partition heals Sentinel is able to have a coherent view of the configuration.

3\. It is able to automatically reconfigure the old master and the other slaves to replicate from the new master.

4\. It is able to work as a configuration provider for clients, with a well defined handshake in order to trigger the reconfiguration of all the clients.

<comment><USER4> A large reason to use Postgres in the first place is because it's a good CP system. Any clustered version is only worthwhile if it preserves consistency.

It's already possible to use more than one Postgres database while not maintaining consistency, but that isn't desirable for most users.

<comment><USER3> > it's a good CP system

PostgreSQL is not clustered by default so it is only _from the point of view of the theory_ a CP system. Most _proper_ CP systems have the ability to provide a limited form of Availability, which is, they are available in the majority partition (this is not "A" of CAP but is a lot better than, if primary is not available, the whole cluster is not available). If you use PostreSQL with asynchronous replication and a best-effort failover solution to provide some form of Availability, you are doing something that makes sense, assuming your product can tolerate the resulting weak consistency properties.

This is a great example on how CAP does not capture certain important semantics of a system. Technically a single primary node is CP but the real world CP systems have better availability than that, or they are useless (as CP systems).

</comment></comment></comment></comment><comment><USER5> Hey everyone, I work for nanobox and on the yoke project. I just woke up and noticed the thread. The interest is appreciated. There are a lot of great questions here, and I'll try to do my best to answer them individually. Please keep in mind that the nanopack cloud-initiative was just announced friday ([https://blog.nanobox.io/nanopack-a-new-vision-for- automated-...](https://blog.nanobox.io/nanopack-a-new-vision-for-automated- infrastructure/)), so we're working feverishly to get documentation, tutorials, and demos available. Any help or contributions would be greatly appreciated.

</comment><comment><USER6> This is inferior to the Manatee design: [https://github.com/joyent/manatee](https://github.com/joyent/manatee)

Funnily enough I ran into this yesterday when Mist was posted and I looked at what other projects were under the umbrella.

<comment><USER5> Much of the architecture of yoke was inspired by manatee. In fact, we've spent the last 2 years working with Joyent on smartos and illumos, so we were heavily inspired by the great engineering behind manatee.

Manatee didn't work for our use cases, which should not be interpreted as "manatee is bad". Here are a few of the reasons we forged a different solution:

\- While node.js is a great language for many things, we didn't want the overhead of running the node.js/v8 runtime alongside postgres. 50M might seem negligible, but with thousands of postgres clusters for our clients every MB adds up.

\- We didn't want the administrative overhead of managing a zookeeper cluster, and instead built the cluster management semantics directly into the yoke project.

\- This may be different now, but at the time manatee was very heavily integrated into illumos/smartos. We really love smartos, but recognize that not all clients are able to use this tech.

<comment><USER7> You guys seem to be creating a bunch of great little tools, but I wish you would spend more time on documentation.

For example, in this case it's not at all clear how the cluster management works, or even what the semantics _are_.

<comment><USER5> I couldn't agree more. We literally just introduced the nanopack cloud- initiative on friday ([https://blog.nanobox.io/nanopack-a-new-vision-for- automated-...](https://blog.nanobox.io/nanopack-a-new-vision-for-automated- infrastructure/)). Bear with us, we'll get there!

</comment></comment></comment></comment><comment><USER8> similar:

[https://spilo.readthedocs.org/en/latest/DESIGN/](https://spilo.readthedocs.org/en/latest/DESIGN/)

[https://github.com/sorintlab/stolon](https://github.com/sorintlab/stolon)

I wonder how they compare to yoke.

<comment><USER9> If I had a penny for every HA postgres solution

<comment><USER10> I'm no Postgres expert, why isn't something like this built in to the product itself?

<comment><USER11> To discourage people from using something broken. You're talking about a database that waited until 2011 to add a SERIALIZABLE implementation because the existing solutions all sucked, and only did it then because of new research (SSI). They have zero qualms about keeping features out of the database, no matter how useful, if they can't be implemented well.

<comment><USER12> Forget serializable, PostgreSQL 7.0 included functional ACID-safe transactions 5 years before MySQL but didn't include LEFT OUTER JOIN ([http://www.postgresql.org/docs/7.1/static/release-7-1.html](http://www.postgresql.org/docs/7.1/static/release-7-1.html)) . That was my first taste of "if you can't do it right, don't ship it."

</comment></comment></comment></comment></comment><comment><USER13> I can guarantee that this cannot guarantee both of:

\- system stays online when any server goes down

\- after confirming a write it will always be returned in corresponding reads following a failover

</comment><comment><USER14> How does this work?

By skimming the installation instructions I deduce that every time a 'write' call hits the primary server, the primary server sends a message to 'monitor' server which sends a message to primary or secondary server(s) to run rsync?

<comment><USER15> It's really not documented well (and the generated docs are a joke) but from what I can gather, it's bog-standard PG replication with a go app to monitor the replication and do automated failover.

The monitor node appears to be there to act as neutral witness node so that a secondary is only promoted leader if it can reach the monitor _and_ the monitor cannot reach the current leader. Hopefully, the leader also commits suicide when it cannot reach the monitor.

MS SQL server implements this pattern and calls it "Database Mirroring Witness": [https://msdn.microsoft.com/en- us/library/ms175191.aspx](https://msdn.microsoft.com/en- us/library/ms175191.aspx)

</comment></comment><comment><USER16> How likely is this to be merged upstream..? There's a glut of reasonable-to- good third-party solutions which I'm always hesitant to commit to as they are just Some Startup's solution.

Not saying it doesn't work for them or it couldn't work for me, but I do feel its justifiable to deduct a few "points" there.

<comment><USER7> Zero chance. For one, it's written in Go. Postgres is written in C and needs to be portable to a lot more architectures than Go currently supports.

Secondly, this is one possible design. The Postgres team prefers tools like these to complement the core so that people have a choice between different solutions depending on their use case, network topologies, availability tolerances, etc.

This just a helper tool that can control Postgres and ship logs around. A native Postgres solution could do it better.

</comment></comment><comment><USER17> no docs no party

<comment><USER18> [https://godoc.org/github.com/nanopack/yoke](https://godoc.org/github.com/nanopack/yoke)

<comment><USER15> That is not documentation, that's a bunch of code comments and signatures turned into a web page. You cannot tell from this documentation how the system works, what are the failure modes and how to recover. Those are kind of useful things in a failover management system.

</comment></comment></comment><comment><USER19> ddddddddddddddddddd</comment>


Show HN: LibSimpleCGI – A straight forward way to write REST endpoints in C++ - <AUTHOR> https://github.com/cjhanks/libSimpleCGI/ <comment><AUTHOR> Hi HN, I am looking for feedback. I wrote libSimpleCGI after years of running into cases where I needed to write C extensions for WSGI apps. Extensions born from either performance concerns OR legacy code reuse.

I was curious if this is/was a common problem for other developers and whether this library is of any practical use.

</comment><comment><USER2> This actually works! Good work.</comment>


UK's gov. believes China could have 40 times more Covid-19 cases than it claims - <AUTHOR> https://www.businessinsider.com/coronavirus-boris-johnsons-government-reportedly-furious-with-china-2020-3 <comment><USER2> Wonder if this overlaps with 21 million Chinese _dropping_ their cell phone service.

[https://www.bloomberg.com/news/articles/2020-03-23/china- s-m...](https://www.bloomberg.com/news/articles/2020-03-23/china-s-mobile- carriers-lose-15-million-users-as-virus-bites)

<comment><AUTHOR> Also perhaps the reason that the Chinese government kicked out so many Western Journalists lately....

<comment><USER3> This is incorrect; Chinese government kicked out __American __journalists to retaliate Trump expelling Chinese journalists that work for state-run news medias [0].

As a China national, I'm disappointed to see Trump's action somehow feeds into CCP's rhetoric.

Sorry for replying to an old comment, saw this while looking at another article. Hope you don't mind!

[0]: [https://www.nytimes.com/2020/03/17/business/media/china- expe...](https://www.nytimes.com/2020/03/17/business/media/china-expels- american-journalists.html)</comment></comment></comment>


Proposal to achieve the "Authentic Political Representation" we dream of - <AUTHOR> http://www.catchandopoliticos.com/index.php?idioma=2 <comment><USER2> Amazing concept and the potential here is monumental and positive.

Upon first impression when visiting the homepage..

The site layout is crowded with text and elements. I would suggest adding more space between some of the elements, and also making the candidate evaluations not take up 100% of their container's width, adding some space on top and to the right and left would make it easier to differentiate between different objects.

Okay, that was enough criticism of the VISUAL aspect of the site, I know this is much more substantive and aesthetic quality is not the priority so I will end it here.

I think perhaps a navigation bar for candidate regions/states/countries would make it easier for constituents to find their candidates.

Awesome website this is high quality stuff.

<comment><AUTHOR> Thank you quadrangularis for your feedback. I really appreciate first of all the concepts and ideas that you wrote down. But also the time that you invested in understand the concept that I'm trying to deliver with this proposal. I understand and agree with all your suggestions. Along with them, the challenge is how to achieve an effective and massively promotion of the initiative. I really believe that the key for this project and many others is collaboration. This idea works based on that concept and also the way to carry it out successfully passes for this same aspect. Thank you so much again for your contribution.

</comment></comment><comment><AUTHOR> I invite all of you to explore this concrete proposal and I'll greatly appreciate any feedback. Many thanks in advance.</comment>


Convore Shutting Down - <AUTHOR> http://blog.convore.com/post/17951919109/convore-shutting-down-april-1st

<comment><USER3> Grove is fantastic. I've been using IRC to communicate with my team forever and we were just using freenode. I finally got around to setting up an IRC server on one of our machines but when it came time to configuring and securing it properly it was going to take days to read through all that documentation, so I just locked it down at the firewall level and we all connect using a forwarded SSH port, or from the local host (we all use irssi as our chat client anyway).

A (rather large) ad agency that a friend of mine works for has a small group of people that wanted a collaboration solution and they all liked IRC so I sold them a monthly VPS with an IRC server configured but guess what - <USER2> have to forward a port to connect. That was cool, they said, they're on OSX and I provided them with scripts that just setup the port and opened Adium for them in a single click ... then along came a contractor that wanted to use windows :S

So I said look: you can pay me to figure out all this shit for you, it'll take me days and cost you thousands of dollars, but for that same money you could buy a few _years_ of a grove.io subscription and I can still connect all the same bots that I've got running for you.

Seriously the price point they've got there is great. Properly configuring and securing a high availability IRC server isn't as simple as "set and forget" you have to do quite a bit of reading and configuring in order to get it running properly. Plus Grove has a bunch of web configuration options that are really handy for less technical folks.

IRC is the chat technology of the past, present, and future, keep it up guys.

<comment><USER4> I agree with the praise for IRC, but I don't see what you need grove for. There's mibbit, irccloud and a few other web IRC clients, all of which seem quite a bit more advanced than grove. And for the server-part; why not hop onto freenode, make a private channel - and off you go?

If you insist on private server then that shouldn't take someone linux-savvy more than 15 minutes either.

You literally type 'apt-get install inspircd', tweak two and a half values in the config file (to enable SSL and set a server-password) - and that's it. An irc-daemon is not exactly maintenance-heavy, mine has been running for over a year without a blip.

<comment><USER3> Firstly, you're not actually allowed to use freenode like that, technically it's supposed to be for open source projects right?

Secondly, I wouldn't run a server application like that on a publicly accessible server without learning pretty thoroughly about how to configure it securely, ymmv obviously but I just wouldn't do it.

Thirdly, there are a bunch of web configuration options that allow non- technical folks to login and do a bunch of stuff. If my client were paying me to do that for them they'd blow their Grove.io budget by an order of magnitude or two each month.

<comment><USER5> So don't use Freenode. EFNet has had invisible, keyed channels since forever.

</comment></comment></comment></comment><comment><USER6> I signed up and tried grove a few weeks ago, I didn't really see the point. I then stumbled across some video tutorials for using hook.io to connect to an IRC server (#4,5 in the list of video tutorials) (<http://blog.nodejitsu.com/hookio-video-tutorials>) and immediately thought of grove.

It seems, to me, that they decided to use the tutorial, hook up web sockets (#9 in the tutorial list), add a nice interface, log all the events hook.io emits, and decided to charge way too much for it.

I could be missing something, but that was my initial impression.

<comment><USER7> If you're comfortable configuring and maintaining your own IRC service and have the time to do it, I daresay you're not Grove's target customer.

<comment><USER8> It's not exactly hard! Just compile a package, run it and forget about it...

<comment><USER9> The same logic could apply for hosting your own git server. And yet, GitHub thrives. Why is that?

<comment><USER10> Also Heroku. Tons of examples...

<comment><USER11> To be fair, unless you're a huge organisation with thousands of users, an IRC server is just a single process on a low-end machine or small VM. You can't really compare that with the whole infrastructure Heroku provides.

</comment></comment></comment></comment></comment></comment><comment><USER12> Paul Graham said. "We're trying to figure out why this YC batch did so well. One theory: they all used Convore (<http://convore.com)>

What actually made the startups switch _away_ from convore? At the point pg said that, quite a number must have been using it. Then they just stopped?

<comment><USER13> There are many reasons why Paul Grahama might speak too highly of Convore, but one theory is endowment effect:

<http://en.wikipedia.org/wiki/Endowment_effect>

People think more highly of things that they have an ownership stake in.

</comment></comment><comment><USER14> So they burnt the Convore customers ... why would I trust them with Grove, exactly?

Pivoting in real life is an elegant move; this feels more like stumble-and- run.

<comment><USER15> Leah Culver can't do startups right. First Pownce, now Convore. What's next? Grove? Let's wait and see.

<comment><USER16> While I'm not a fan of Leah Culver at all (I view her as Eric Raymond^2, better at selling the image of geekiness than at actual coding), most startups fail even with good founders.

Besides, don't we pride ourselves on accepting failure and treating it as a learning experience?

<comment><USER17> I've met her a couple times, and have a different opinion. (I admit, I was biased a bit before meeting her, based on all the press and the murder of a friend's RAID while drunk...)

While she might not be the best computer scientist in the world of startups, she's great at the other roles in a startup, and is certainly a competent developer now. Not everyone in a startup needs to be a classically trained computer scientist with a Stanford PhD. Not every successful startup has to look like Google. You know who also wasn't a great computer scientist? Steve Jobs.

(I actually met esr when I was growing up in Pennsylvania, too -- he ran the local community Free-Net which I used for Internet access. He was a hardcore geek back then, and seems to have shifted more toward self-promotion and politics over time. He actually wrote some decent code for BSD/OS and ISP operations.)

</comment></comment></comment></comment><comment><USER18> Late last year I was in an incubator where Convore started. There were 2 other companies doing the same exact thing .. All of the YC companies were using HipChat, Convore seemed like "yet another copycat site to sign up for an account with". Chat is a solved problem. Let's move on.

<comment><USER19> Nothing is a solved problem. HipChat's a great product, and in my opinion the current market leader, but there are always ways to innovate and disrupt. (That being said, using IRC probably isn't it.)

<comment><USER20> You can't mean this literally.

<comment><USER19> No, I do mean it literally. The wheel is a pretty basic invention, and yet tire manufacturing is a $140 billion industry that spends a great deal on research and development.

When you think a problem is solved, it's actually in a perfect position to be disrupted.

<comment><USER21> Are you familiar with the phrase "reinventing the wheel"?

<comment><USER19> Yes. My point was that there's a huge amount of effort currently being expended to do just that. No one is trying to change the fact that a wheel is round and rolls, but from there, there's quite a bit of space to innovate.

</comment></comment></comment></comment></comment></comment><comment><USER22> My opinion when I got accounts on Convore and Pownce: "This is neat, I guess."

I can't quite articulate what I think they did wrong... but these products just felt too startupy to survive.

<comment><USER23> Agreed. But their "pivot" into Grove is actually really valuable. Chat is pretty essential for teams (ours at least), but it's not that easy to build it yourself without running into major scope creep and other headaches. I'd rather pay a little for reliability rather than distract the team from the main goal to build a chat system.

</comment></comment><comment><USER24> I would have preferred to see an extended read-only period, rather than a month in which to panic-download your data (and the export functionality seems to be broken right now too <https://convore.com/export> ).

I'd really like it if there was more time to capture the site and submit copy to the internet archive. I'll see what I can do with archive team.

So long, and thanks for all the data! Perhaps convore is an apt name for something that consumes your history. :-)

<comment><USER25> Export worked for me, but it's moronic. It seems like every user gets all his utterances.. which are utterly useless without context.

They're basically treating the data as if it's email or facebook posts and comments (those last also mostly useless without context). But it's not, it's a far finer granularity of interaction.

Convore, please let users download logs for the channels they're members of.

</comment></comment><comment><USER26> I'm working on a site with functionality similar to convore. Discussion is centered around links in a manner similar to reddit or HN. Just finished an alpha test, you can register for the beta at <http://b00st.com>.

</comment><comment><USER27> I don't understand Grove. Why would you pay so much for hosted IRC with very limited users? $10/m for 5 users seems crazy high to me.

<comment><USER28> Especially since Skype gives you all of their functionality for free. I'm really wondering what advantage this has over Skype group chats (which is free).

<comment><USER29> Without even looking at their page: logging, persistence, availability, API hooks.

Skype is awesome and is the cornerstone of my team for private communications. But if history is only transferred when parties are connected, and only between people who are explicitly added to each chat, it leaves a giant gaping hole in group communications.

<comment><USER4> _logging, persistence, availability, API hooks._

Skype has all that, except the API hooks (which I'd expect pretty low on anyones list).

Also not sure what you mean by "availability". Skype is about as available as it gets - Everyone and their dog already has it installed.

<comment><USER29> Availability means it's online all the time. A server isn't, people's Skype isn't. Maybe it works for you if you're all in the same office, but I have people in 8 different time zones.

</comment></comment></comment></comment></comment><comment><USER30> Im glad the team is moving on, bigger and better! But why not keep the site up. How expensive could it be to show your commitment to the few users you do have. If anything it builds a good reputation on keeping your projects going even if it wasn't the massive success the founders wanted. Im all for progress but every project doesn't need to die does it?

<comment><USER31> Even open sourcing it would be good enough for those that really care to self- host.

</comment></comment><comment><USER19> I never quite got Convore. Grove is sort of interesting, but I doubt it'll displace us from HipChat.

The major advantage of using a non-proprietary protocol like IRC is that people can choose their own clients... but the state of most IRC clients nowadays is abysmal. Colloquy on OS X, for example, hasn't been updated in years.

If I was going to use IRC to communicate with my team, I'd just create a private channel on Freenode. We'd miss out on archiving, but I don't think that that feature alone is worth $2/user/month.

<comment><USER32> Freenode is for free software. It's not "free as in someone else runs our private chat for us." Hosting business chat via terms of service abuse is inadvisable to say the least.

<comment><USER33> Many companies have used private and/or secret channels on efnet for decades without complaint.

</comment></comment></comment><comment><USER34> I still get a weekly email from Convore telling me about new posts in the groups I belong to, and when I try to click through to them, I get sent to arbitrary pages. How can one unsubscribe from the mails?

<comment><USER35> You can unsubscribe from all Convore email here: <https://convore.com/settings/notifications/>

Just make sure that you sent "Send me summary e-mails" to "Never". Hope that helps!

</comment></comment><comment><USER36> Honest question: why would I use Grove over HipChat?

<comment><USER37> It's not really about Grove vs HipChat but more IRC vs HipChat. IRC is a really old and open protocol, meaning that hackers built lots of tool around it and already love it. But it's a pain to configure and maintain it if you've got more important things to do. (Like building your app). A couple years ago, I believe grove wouldn't have stand a chance against the hacker mentality of "I'll do it myself", but now that more and more companies need to go as fast as possible, it makes sense to pay a little bit to have a service that's stable and just work out of the box.

And, for me personally, I hate HipChat clients (IIRC they run on a crap adobe version?) and I like using the existing tools I have for IRC. For instance, I've already setup everything to work on my vps with irssi, with the plugins I want, the theme I want, etc. I don't want a "new and prettier" solution to a solved problem.

<comment><USER38> I'd say it's about "the chat tool you use" vs grove. In my case, it's also HipChat/Campfire.

Unless you're focusing SPECIFICALLY on "hackers that love IRC" - in which case your market is quite small (compared to the whole corporate chat market), imho...

<comment><USER37> Well, "hackers that love IRC" is still a huge market if you think about all the software companies. And, even if it's a smaller one, sometime it's better to create a product that'll be used by less people rather than trying to build something for everyone and ended up with nobody using it. But I feel your concern, Grove might become a successful small business but they'll have to attack other market or build different products to continue to grow.

</comment></comment></comment></comment><comment><USER5> Looks like they gave it a year: [http://eflorenzano.com/blog/2011/02/16/technology-behind- con...](http://eflorenzano.com/blog/2011/02/16/technology-behind-convore/)

I'm not trying to dance on their grave or anything, but from reading this article it appears a company I worked with last summer had read it as well, and that level of -- let's call it "sophisticated" -- messaging architecture was _beyond unwieldy_ for an early-stage company. I can understand wanting to avoid Michael Arrington calling you "amateur hour" when things fall over under popularity, but there's a such thing as overengineering.

I realize my opinion might be based on a red herring or strawman, but when chat is the "Hello World" of server-side JS I'm just not sure there was enough _outside of the architecture_ to hang a company on.

</comment><comment><USER39> Grove looks really interesting.

The pricing strikes me as a bit odd though.

You're actually charging more per user for more users ($2 per user for 5 users; $2.50 per user for 50 users), rather than giving larger accounts a discount. I imagine the reason for it is the load it places on your infrastructure, but I've rarely seen that kind of price stacking work in the wild.

<comment><USER35> Past the first plan, the pricing is all $2.50 per user. I like to think of the first plan as the "startup" plan so we made it cheaper. We do offer discounts per user past 50 users.

</comment></comment><comment> no-espam I still dont get who would use grove? IRC users aren't exactly looking for a client replacement. Existing IRC clients just work, like Craiglist. Colloquy on the Mac, etc.

</comment><comment><USER40> irc ? 2012 ? .................................

<comment><USER21> Hah, that's kind of what I thought. I think people must be secretly jumping on their gaming servers.</comment></comment>


Updated 2017 BMW i3 goes 114 miles with new 33-kWh battery - <AUTHOR> http://www.autoblog.com/2016/05/01/updated-bmw-i3-114-miles-new-33-kwh-battery/ <comment><USER2> I'm sorta within the target audience of the i3. I live in a mostly walkable mid-sized European city with a comparatively well implemented public transportation system. Nonetheless, and particularly in the colder months, I would like a small electric city car.

Unfortunately the i3 is almost too much car. For my intended use it's pretty expensive and kinda big... at least compared to the Renault Twizy. The Twizy is much closer to the size and cost that suits my probable usage but it's obviously designed as primarily a summer car and it's honestly right at the lower of limit of acceptable size. If Renault chose to update the battery pack for the Twizy it would be a much more attractive option. (For those who might not know, Twizy owners do not buy or own the battery, they must sign up for a long term lease / maintenance contract).

I keep hoping someone brings a vehicle to market that's in between the BMW i3 and the Renault Twizy... or for some reason a discounted i3 appears on the used market in my area.

<comment><USER3> I think the i3 has great potential and mass appeal and is accessible at its price point. But having said that, I'm still confused why BMW went out of its way to design it as an "alternative car." I might be alone in this, but I find the car really ugly. But it wouldn't stop me from driving one!

<comment><USER2> Yeah, it's not really an attractive car is it! Though having sat in one briefly I have to say that, given the external footprint, the interior is enormous and those little suicide doors make it feel even bigger.

</comment></comment></comment><comment><USER4> For 30K more (a lot I get but...) you get 2-2.5x the range and a way cooler car for the lower end S.</comment>


Without Gizmodo, The 4G Would Still Be "Stolen" - <AUTHOR> http://judstephenson.com/2010/05/04/without-gizmodo-apple-would-still-be-without-their-beloved-iphone-4g

<comment><AUTHOR> Main Point: Diving a little deeper into the law, you can see that in California, a legal defense against receiving stolen property is claiming innocent intent. Simply put, you can't be convicted of a California Penal Code 496 charge if you intended to return the property to its owner or the police when you bought or received the stolen property.

Think about it another way, the only chance that Apple had for getting the phone back (if the person who found it didn't give it back), was for Gizmodo to buy it before another, less well intentioned buyer emerged (think chinese knockoff factory).</comment>


Fundable Acquires Clarity.fm as Part of Its New Launch Platform, Startups.co - <AUTHOR> http://techcrunch.com/2015/02/03/fundable-acquires-clarity-fm/ <comment><USER2> Is there any way to tell within Fundable how much equity the startup gave up in order to attain their funding goal? Referring to these examples: [https://www.fundable.com/raise-capital](https://www.fundable.com/raise- capital)

<comment><USER3> There's not.

</comment></comment><comment><USER4> Why choose a ".co" instead of an entirely different name or TLD? Isn't .co quiet confusing? I think its a bad name inspite of the fact that most people come to you via a search engine.

<comment><USER5> Because it doesn't really matter anymore, it seems. Lots of established sites are on alternative TLDs now or at least _began_ on them. I imagine the number of people who type in full domains to access a site each time nowadays without autocomplete coming up is very low.

<comment><USER6> I think "it doesn't really matter anymore" is right on the money.

.co as a TLD was essentially built around tech companies and directly marketed to them.

Plus if we're honest - most of the time people Google for companies websites that don't autocomplete in their address bar...

As an OTT example: If I want to come to Hacker News I type HN into Google and its the top result. I type two letters and click one link vs typing 'news.ycombinator.com' (typing 'news' for me brings the BBC). If a startup has put thought into their branding then a full web address isn't needed to find them. For that reason .co isn't confusing to most web users these days.</comment></comment></comment>


Show HN: Send HTTP POSTs to Your Inbox – PostTo.Email - <AUTHOR> https://postto.email/ <comment><AUTHOR> Midway to building a much larger project that I may never finish, I decided to extract all of the simple stuff that I was implementing, strip the complexities like authentication and query languages, etc.

So here is a simple, free, non-tracking, and hopefully _stable_ HTTP form handler that emails you each time you get a new submission.

</comment><comment><USER2> I just tested this and it actually seems too easy. Is there a way to edit previously made forms? How do I login to see form submissions?

What is next for this? Seems like a great tool.

<comment><AUTHOR> Currently form submissions are only emailed to you, I will store them in the database if you tick the "Store Submissions" box, and in the near future you will be able to browse submissions.

There is currently no way to edit a form, and I'm not sure if I will ever build this, since it's all semantic, and you can always create a new form if you want.

Roadmap:

\- Request links to form/submissions

\- Export submissions

\- Expose an API to query the submissions for a form (useful if you want to add a comment system to your static blog/website)

\- Form Builder -- I know these are a dime a dozen, and I might just implement one off the shelf so I don't have to reinvent the wheel.</comment></comment>


How will life change post-Covid? - <AUTHOR> <comment><USER2> If you live in Bangladesh then Corona within the country will not be an issue, population of 165M with 17 Corona deaths registered. Median age is 27.6 years by the way.

Trade with other countries probably will be affected in a big way.

</comment><comment><USER3> you should rename it to

       Ask HN: How will life change post-Covid?</comment>