Florida accused of concealing worst tuberculosis outbreak in 20 years - <AUTHOR> http://www.rawstory.com/rs/2012/07/08/florida-accused-of-concealing-worst-tuberculosis-outbreak-in-20-years/

<comment><USER2> Please do not link a news aggregator, the Palm Beach Post was the creator of the story and should get the traffic. ([http://www.palmbeachpost.com/news/news/state- regional/worst-...](http://www.palmbeachpost.com/news/news/state- regional/worst-tb-outbreakin-20-years-kept-secret/nPpLs/))

</comment><comment><USER3> _Believing the outbreak affected only their underclass, the health officials made a conscious decision not to not tell the public, repeating a decision they had made in 2008, when the same strain had appeared in an assisted living home for people with schizophrenia.

"What you don't want is for anyone to have another reason why people should turn their backs on the homeless," said Charles Griggs, the public information officer for the Duval County Health Department._

</comment><comment><USER4> >Believing the outbreak affected only their underclass, the health officials made a conscious decision not to not tell the public

I love the part where homeless people aren't considered part of "the public".

<comment><USER3> Of course they are. Homeless people were not informed either.

<comment><USER4> The implication is that it's ok when homeless people get TB. An outbreak that is confined to that "underclass" should only raise alarm when it infects people that actually matter.

And the number of cases isn't the metric used. Imagine if the TB outbreak only effected owners of yachts? There's certainly fewer yacht owners than there are homeless people!

<comment><USER3> I know it sounds like class warfare but they're actually trying to prevent that. _Believing the outbreak affected only their underclass, the health officials made a conscious decision not to not tell the public, repeating a decision they had made in 2008, when the same strain had appeared in an assisted living home for people with schizophrenia.

"What you don't want is for anyone to have another reason why people should turn their backs on the homeless," said Charles Griggs, the public information officer for the Duval County Health Department._

Don't know if it's true, but it's at least plausible.</comment></comment></comment></comment>


Six percent of online adults use reddit (15% of young men) - <AUTHOR> http://pewinternet.org/Reports/2013/reddit/Findings/Six-percent-of-online-adults-use-reddit.aspx <comment><USER2> Note this survey was carried out in America: the conclusions are not valid for adults of other nationalities.</comment>


TPP Text Confirms Massive Loss to Canadian Public Domain - <AUTHOR> http://www.michaelgeist.ca/2015/11/official-release-of-tpp-text-confirms-massive-loss-to-canadian-public-domain/ <comment><USER2> We need a massive, concerted effort to widely distribute digitally, for free, the works of all the authors who would have entered public domain. It's clear that negotiating with our leaders has failed; the free and open internet that has served us so well in the preceding decades is closing faster than ever and our only hope is digital civil disobedience.

<comment><USER3> Whilst "digital civil disobedience" is all well and good, breaking their rules won't help us break their will.

The last bastion of hope is a "union of the consumers", where we collectively punish and break companies that we collectively see as dangerous to the future of our world.

We only need to tear down one company to be able to set off a chain reaction. Once consumers realise the power they hold as a well organised and effective collective, the people can pull back some of the power from the corporation (and our governments to which they now hold the power over).

Corporations can't survive without income and customers. Every day we vote with our wallets without giving it a second thought. It's time we started to appreciate that fact.

Edit:

I know this isn't particularly realistic :-(

<comment><USER4> How will this work practically? Let's say we want to try this just for a month. Make a list of such companies and try not to spend our money with them. First of all, making this list itself isn't that simple, because of confusing and ever changing murky corporate structures (apps like buycott [1] can help here). In many industries, a handful of players control the entire market - just 3 or 4 companies control meat supply in the US, Monsanto controls a _huge_ portion of seeds and so on. In the end we'll probably end up with a list of couple of dozen companies in our "bad list" and our options to shop (even the basic necessities) will be severely limited. Not saying it can't be done, it is just difficult to do it without some serious _collective_ effort. It is definitely a good exercise to brainstorm ideas though, we gotta start somewhere.

[http://buycott.com/](http://buycott.com/)

<comment><USER5> You buy from the least bad one.

Then the others become less bad, and you switch to them. And so on.

<comment><USER6> I don't think that's feasible either on a large scale. How about this strategy:

1\. Pick the top three offenders (not in the same industry).

2\. For each offender, find their cash cow, or top 3 most profitable products.

3\. Make a concerted effort to boycott those products, and those products alone, indefinitely. Make some real damage.

The purpose here is to make an example out of someone. Build morale among protesters that if well organized, they can be effective.

First scenario: One million US/Canadian citizens sporadically decide to buy from company Y instead of company X... nobody notices.

Second scenario: Company Z suddenly loses 100,000 customers for their core product... I'll bet you someone notices.

<comment><USER7> Yes, see the history of fossil fuel divestment and boycotts, [https://en.wikipedia.org/wiki/Fossil_fuel_divestment](https://en.wikipedia.org/wiki/Fossil_fuel_divestment)

</comment></comment></comment></comment></comment></comment><comment><USER8> Copyright does no longer exist to protect creators, but to protect the distributors. We badly need a reform.

<comment><USER9> Isn't distribution done by third-party services either? Kindle store or iTunes store aren't copyright holders, for example.

The problem with copyright is that it is used as an excuse to deliver the good idea to protect the new authors who are still alive, in the same package with the bad idea to let unspecified number of loosely related and unrelated people feed off free money source created by long dead authors decades ago.

<comment><USER10> > _The problem with copyright is that it is used as an excuse to deliver the > good idea to protect the new authors who are still alive, in the same > package with the bad idea to let unspecified number of loosely related and > unrelated people feed off free money source created by long dead authors > decades ago._

I think part of the idea is that whether the creators are dead or not, having copyright and "unrelated people feed off", still maintains an economy over the item and gives it monetary value. Whereas if it was de-copyrighted it would lose that value -- everybody could just copy it.

E.g. Disney only being able to create or allow the creation of Mickey Mouse stuff (t-shirts etc), leads to a large business for them, taxes for US etc. (Plus pays a lobby to ensure longer copyright). If Mickey Mouse was suddenly public domain, that would stop and its characters would be devalued very fast, with everybody competing with ever cheaper products with him on.

<comment><USER11> And they would come up with some other creative character with a new copyright duration, thus enriching our public culture with their previous works and employing people to create new works for people to enjoy.

Twenty years ought to be enough to recoup your investment, but I'm open to compromise. In any case, (rougly) 150 years, as it currently is, is way, way, way, way, way, way way too long.

<comment><USER12> Way too long for why? I don't believe citizens are harmed by not being able to make money from something they didn't create.

<comment><USER11> Really? Copyright is siloing off culture in unprecedented ways. Martin Luther King's "I Have a Dream" speech is under copyright by his estate, and will be until the 2040s. It can't be used by anyone without paying (large) licensing fees to his estate, even though they had literally nothing to do with the creation of the speech.

Ever notice how restaurants and movies never use "the Happy Birthday song"? That's because Warner Bros claimed copyright on it for decades. Another part of our culture that can't be used thanks to copyright.

There are plenty of films, plays, and TV series based on works in the public domain. Romeo and Juliet, Pride and Prejudice, and Sherlock Holmes to name just a few that come to mind. Should those creators be required to pay tribute to the descendants of the long-dead authors in order to use those parts of our shared culture?

<comment><USER13> A famous stained-glass window in a church in my home town was heavily censored (i.e. publishing photographs of it resulted in legal threats, fair use or not) because the estate of the late artist who created it vigorously enforced his copyright. High-resolution photographs were available for free of every stained-glass window on the church website, except for this one. At times there were even talks about covering it up physically because of claims about public display licensing issues. The artist in question likely didn't see any of this coming during his lifetime.

Being granted exclusive protections during your lifetime or for a reasonable period of time after the creation of a work is defensible. De facto perpetual protections afforded to the artist's grandchildren (or great^n-grandchildren), not so much.

</comment></comment></comment></comment></comment></comment></comment><comment><USER14> Is there any indication that this has an overall limit of 95 years as per the U.S. Copyright? ("Copyright protection for works published prior to January 1, 1978, was increased by 20 years to a total of 95 years from their publication date."[0])?

This cannot be stressed enough, that the real losers here aren't people who want to remix ultra-famous copyrighted works-- it's that works that nobody cares to profit upon will remain orphaned.

If there is one protection to the Public Domain that needs to take place, it's that copyright owners should have to show a willingness to profit off their material (by paying a recurring fee) so we don't experience this deadweight loss.

[0] [https://en.wikipedia.org/wiki/Copyright_Term_Extension_Act](https://en.wikipedia.org/wiki/Copyright_Term_Extension_Act)

<comment><USER14> More on Orphaned Works:

"Significant among those changes were the elimination of the registration and notice requirements, which resulted in less accurate and incomplete identifying information on works, and the automatic renewal of copyrighted works that were registered before the effective date of the 1976 Copyright Act." [0]

Bringing back registration and ending automatic renewals would do a lot of good. (Adding a fee would be better.) Has there been any work on laws to do this?

[0] [http://copyright.gov/orphan/reports/orphan- works2015.pdf](http://copyright.gov/orphan/reports/orphan-works2015.pdf)

<comment><USER15> The problem is the Berne convention explicitly forbids requiring registration for a work to enjoy the benefits of copyright. The USA ratified this convention in 1989.

However, I agree with you. There needs to be some mechanism pushing back on copyright holders encouraging them to let their works revert to the public domain. A nominal fee at regular intervals would go a long way to encouraging this. However, perhaps something more aggressive is needed. Since the advocates of increased copyright duration often phrase their argument in terms of "intellectual property", perhaps a copyright property tax would be more appropriate.

"OK Disney, you can keep the mouse out of the public domain, but that will be $50M/yr."

<comment><USER14> Thanks for this pointer.

At this juncture, it certainly seems that the US shouldn't have caved to signing Berne. We made it 100 years, why didn't we stick it out?

Well, commendations to Zoe Lofgren (representing San Jose) at attempting to renew U.S. copyright formalities. [0]

[0] [https://en.wikipedia.org/wiki/Copyright_formalities#Renewed_...](https://en.wikipedia.org/wiki/Copyright_formalities#Renewed_interest_in_formalities)

EDIT: It appears she hasn't pushed ahead copyright formalities (in the form of the Eldred Act) much since 2005. Here's hoping something new comes up...

</comment></comment></comment></comment><comment><USER16> Infinite copyright terms are unconstitutional in the United States. Hopefully SCOTUS will notice that someday.

<comment><USER17> Have the copyright extensions actually been upheld in court? Nothing stops a legislator from passing an unconstitutional law, but if nobody fights it then it won't get struck down.

<comment><USER18> > To promote the Progress of Science and useful Arts, by securing for limited > Times to Authors and Inventors the exclusive Right to their respective > Writings and Discoveries.

Depends on how you literally minded you read it and how the Supreme court feels:

1) Any copyright or patent that does not promote Science of Useful arts is unconstitutional

2) Only written words are protected, not visual or software.

3) There should be some limitation on the term but it could be way past couple of googols in years.

and so on. The constitution is (un) intentionally vague in a lot of places.

<comment><USER19> You forgot

4) Copyright cannot extend past the life of the author, since only the author can have exclusive rights to his writings. I.e. copyright is non-transferable to heirs, corporations, etc.

<comment><USER20> Do you think I'm being paranoid if I thought that the author/copyright holder would suddenly have an "accident" if their death meant it would end the copyright?

</comment></comment></comment></comment></comment><comment><USER21> Public Domain's Number One Enemy = Mickey Mouse. Until there is some way to protect Disney and not have its IP placed in public domain we will never have nice things again in the public domain.

<comment><USER22> But ... this is wrong. Mickey should be in (or be headed towards) the public domain.

<comment><USER21> Until there is a solution for Mickey Mouse the millions of dollars will keep everything behind copyright. We solve that one we will have Public Domain again.

<comment><USER22> We solve that one and everyone follows Mickey's lead and nothing enters PD.

</comment></comment></comment></comment><comment><USER7> If you want to do something positive:

1\. Organize a local event or travel to DC in the week of Nov. 14, there is a 90-day window to mobilize opposition to TPP, [https://www.eff.org/event/global-week-actions-against-tpp- tt...](https://www.eff.org/event/global-week-actions-against-tpp-ttip-tisa). EFF actions in DC will coincide with APEC meetings in the Philippines, where US officials (including trade reps) will push the TPP, and EFF Philippine allies will have mass protests.

2\. Remember how many companies rallied against SOPA? Ask your tech employer to issue a public statement on TPP and digital rights. Create a public list that links to company statements. TPP unfairly advantages legacy business models at the expense of emerging business models. It is not about "elites" vs. "citizens", it is about crony insider companies vs. all other companies.

3\. Ask organizations like Wikipedia, Archive.org, Google, Facebook to run banners to raise awareness about TPP, asking users to flood TPP governments with calls for representation. Companies and citizens have never had so many channels for rallying opposition. If TPP (and TTIP and TISA later) passes, those channels may be limited in the future. Use them or lose them. We not only need to stop each of these bad agreements, we need to reduce incentives for future bad agreements. They are a DDoS on the legislative process, at the expense of opportunities to create sensible rules which support technology innovation and the goals of modern civil society.

</comment><comment><USER23> I put up most of the TPP on genius.com yesterday [0]. All chapters from 0 to 30 are there except for chapter 20 and the annexes of chapters 2, 12, and 15. I am still working on reformatting the text since formatting is lost when copying from a PDF to an HTML textbox.

I still have not heard back from genius.com support about some of the technical issues I am having: I can't delete/move chapter 20 with the rest, and new chapters do not appear in the collection list ("album" in genius parlance).

If you know anyone with legal expertise who is willing to annotate a few places, please forward them to [0]. Alternatively, please retweet [1]. If you want to read the US government's take on the TPP, this link has a summaries before the full text in each of the chapters [2].

[0] [http://genius.com/albums/Transpacific-partnership-tpp- negoti...](http://genius.com/albums/Transpacific-partnership-tpp-negotiating- parties/Trans-pacific-partnership)

[1] [https://twitter.com/TeeAyKay/status/663830955971338240](https://twitter.com/TeeAyKay/status/663830955971338240)

[2] [https://medium.com/the-trans-pacific-partnership](https://medium.com/the- trans-pacific-partnership)

If you've made up your mind about the TPP in whichever direction, please let your government know (Use #TPP on Twitter):

Canada: Prime Minister's Office: [http://pm.gc.ca/eng/contactpm](http://pm.gc.ca/eng/contactpm)

Prime Minister's Twitter: [https://twitter.com/JustinTrudeau](https://twitter.com/JustinTrudeau)

Minister of International Trade: [https://twitter.com/CanadaTrade](https://twitter.com/CanadaTrade) and [https://twitter.com/cafreeland](https://twitter.com/cafreeland)

Members of Parliament (by postal code!): [http://www.parl.gc.ca/Parliamentarians/en/Constituencies/Fin...](http://www.parl.gc.ca/Parliamentarians/en/Constituencies/FindMP)

USA: White House: [https://www.whitehouse.gov/contact/write-or- call](https://www.whitehouse.gov/contact/write-or-call)

White House: twitter [https://twitter.com/WhiteHouse](https://twitter.com/WhiteHouse)

House: [http://www.house.gov/representatives/](http://www.house.gov/representatives/)

Senate: [http://www.senate.gov/general/contact_information/senators_c...](http://www.senate.gov/general/contact_information/senators_cfm.cfm)

* I am not affiliated with genius.com, or any political party. I just want an informed decision and the best possible outcome, whatever it may be.

Edit: spacing

</comment><comment><USER24> What the hell is the justification for the document being secret, anyway?

<comment><USER25> See the discussion on another link: [https://news.ycombinator.com/item?id=10532565](https://news.ycombinator.com/item?id=10532565)

Summary: international trade deals involve a lot of tradeoffs in various sectors of a country: e.g. throwing textiles under the bus in order to get some advantages in automotive. If the initial negotiation of the treaty was open, special interest groups would make massive amounts of noise and no country would be able to come to agreement. Once the treaty is done being written, then countries can debate it internally, in its entirety. There will still be lots of noise but at least there is a complete treaty to argue over.

See also: [http://www.npr.org/sections/money/2015/06/26/417851577/episo...](http://www.npr.org/sections/money/2015/06/26/417851577/episode-635-trade- deal-confidential)

<comment><USER26> But that argument stops working once certain industry groups do get access to the negotiations, which I thought was the case here.

Also, this agreement goes way beyond a simple trade deal; parts on this undermine the very basis of our democracy and justice system. I don't see how any democratic country can possibly agree to this.

<comment><USER27> Not only our democracy and justice system, but the systems of the other signatory countries as well. For many of them, the TPP will result in loss of relative sovereignty to the American multi-national corporations that are privy to the details of the deal-- Novartis being the biggest pusher in the pharma aspects of the deal as they relate to copyright.

<comment><USER26> Not my democracy and justice system, actually. I'm more concerned about TTIP in that respect.

</comment></comment></comment></comment></comment><comment><USER28> A while ago, I said something like "the elites have totally won the battle against the slave class (formally known as the middle class)" to a friend and he was shocked that I felt that way. I was half way joking but the more I think about it the more I think that it is game over for the non-elites, except for a few things that people can do:

1\. Pay a lot of attention to your own job skills and education, managing career, etc.

2\. Put a lot of energy into maintaining your economic outlook, that of your family and friends, and that of your local community

3\. Ignore crap like the show of elections, and the show of "news" in general. If you want to do something positive then pick a cause, like a constitutional amendment to overthrow the Citizen's United ruling, and work on that cause year after year. Time better spent.

<comment><USER29> It's never been "game over" in the past. History has shown that oppression of the masses is usually the forerunner to a revolution. Why would be any different this time around?

(I'm not advocating revolt as a solution; just questioning the validity of the idea that the 'slave class' have no way to change things.)

<comment><USER30> People really harp off about 1984 a lot when speaking about surveillance but they always seem to leave out what I considered the most disturbing part of the book. The government in the story was very aware that they were following a pattern that would lead to a revolution. So they created a system where revolution was impossible.

1984 like most science fiction, was created to show an extreme. While there is no such thing as an unbeatable system, there is definitely people higher up aware of the possibility and actively preventing any major paradigm shifts in politics. The only thing that would truly force a revolution (which doesn't even have to be violent) is if people don't have enough food or their life is on the line. We saw this in the Arab Spring with the rise in wheat prices, which Saudi Arabia was able to avoid through subsidies.

</comment></comment></comment><comment><USER31> I think the text needs a provision to roll it back gracefully. For instance, by tapering any repercussions in a certain time-window.

</comment><comment><USER32> The Canadian copyright length is being extended. Does the TPP also put a cap on the US copyright length or are countries still free to extend their copyrights as they see fit?

<comment><USER33> It just requests a minimum length, nothing about maximum lengths. (Obviously, as the US needs to extend Disney's copyright soon again)</comment></comment>


Switchblade - Lethal Explosive UAV Drones - <AUTHOR> http://www.latimes.com/business/la-fi-kamikaze-drone-20120611,0,3893056.story

<comment><AUTHOR> So, I'm the one who posted the LA Times article link to HN. This UAV fascinates me to no end. It seems these are launched by the solder who essentially carries this thing around with them until needed. They launch the UAV and then fly it to their target relying on GPS, maps, and most importantly video. They will essentially see the view of the UAV up until the moment of impact whether it is a vehicle, building, or person. I'm sure this thing basically flies itself, following the target via video analysis, and there are tons of ways to abort if needed. How amazing is this? And expensive I bet.</comment>


Missing the point about suggested searches - <AUTHOR> http://arstechnica.com/media/news/2009/05/google-to-searchers-would-you-like-a-torrent-of-that-show.ars

<comment><USER2> Makes me happy. Not because I need it to suggest torrents for me, but it makes me feel like I'm talking to something unbiased, except where indicated. (Of course, that could be a clever ruse...)

I must say though, I'm increasingly irritated at Google's overly 'helpful' suggest in other ways. They've gone too far in the "auto" direction. When I'm searching for technical things (like, all day while programming) the stemming and word substitution can be irritating. I find myself longing for the days when they told you "AND was unnecessary" because they always make sure all terms are in the page. Doing +a +b +"blah" is a bit tedious.</comment>


New PaaS - Write HTML5, Deploy in iOS + Android + Chrome - <AUTHOR> http://trigger.io/

<comment><AUTHOR> Just discovered Trigger.io - a YC backed startup which is a new service to deploy your HTML5 apps across multiple platforms. I just signed up and am checking it out & am trying to figure out how it compares to PhoneGap. Besides the obvious difference of free vs. monthly service charges.</comment>


Ask HN: Where do hardware engineers hang out online? - <AUTHOR>

This is a great place to talk with software developers, is there any equivalent online site for hardware engineers? <comment><USER2> Adafruit, Make Magazine, and Hackaday are some of the hardware hacker hubs.

<comment><AUTHOR> Thanks! I am checking these sites now:

<http://www.adafruit.com/blog>

<http://www.facebook.com/makemagazine>

<http://www.flickr.com/groups/hack-a-day/>

</comment></comment><comment><USER3> <http://electronics.stackexchange.com/>

</comment><comment><USER4> <http://www.electronics-lab.com/blog/>

</comment><comment><USER5> edaboard</comment>


Why HN was down - <AUTHOR> Hacker News was down all last night.  The problem was not due to the new server.  In fact the cause was embarrassingly stupid.
On a comment thread, a new user had posted some replies as siblings instead of children.  I posted a comment explaining how HN worked. But then I decided to just fix it for him by doing some surgery in the repl.  Unfortunately I used the wrong id for one of the comments and created a loop in the comment tree; I caused an item to be its own grandchild.  After which, when anyone tried to view the thread, the server would try to generate an infinitely long page.  The story in question was on the frontpage, so this happened a lot.
For some reason I didn't check the comments after the surgery to see if they were in the right place. I must have been distracted by something.  So I didn't notice anything was wrong till a bit later when the server seemed to be swamped.
When I tailed the logs to see what was going on, the pattern looked a lot like what happens when HN runs short of memory and starts GCing too much.  Whether it was that or something else, such problems can usually be fixed by restarting HN.  So that's what I did.  But first, since I had been writing code that day, I pushed the latest version to the server.  As long as I was going to have to restart HN, I might as well get a fresh version.
After I restarted HN, the problem was still there.  So I guessed the problem must be due to something in the code I'd written that day, and tried reverting to the previous version, and restarting the server again.  But the problem was still there.  Then we (because by this point I'd managed to get hold of Nick Sivo, YC's hacker in residence) tried reverting to the version of HN that was on the old server, and that didn't work either.  We knew that code had worked fine, so we figured the problem must be with the new server.  So we tried to switch back to the old server.  I don't know if Nick succeeded, because in the middle of this I gave up and went to bed.
When I woke up this morning, Rtm had HN running on the new server. The bad thread was still there, but it had been pushed off the frontpage by newer stuff.  So HN as a whole wasn't dying, but there were still signs something was amiss, e.g.  that /threads?id=pg didn't work, because of the comment I made on the thread with the loop in it.
Eventually Rtm noticed that the problem seemed to be related to a certain item id.  When I looked at the item on disk I realized what must have happened.
So I did some more surgery in the repl, this time more carefully, and everything seems fine now.
Sorry about that. <comment><USER2> Amazing that such a large percentage of debugging involves determining exactly _what_ you are debugging. The definition of the problem, many times, is the solution.

Might be a good time to mention Rubber Duck Debuggging. <http://en.wikipedia.org/wiki/Rubber_duck_debugging>

<comment> z-e-r-o For me, this should be called stackoverflow debugging. I genuinely solved a lot of my problems by trying to write a _good_ question on SO about my problem. The problem seems really difficult when I try to ask it in one sentence, just out of my head. However once I try to describe the background, what I'm trying to achieve, what I'm using, when does the problem happen, simplified down to sub-cases, usually by the time I'd be 80% ready with writing the question, I realize the answer.

<comment><USER3> That happens to me a lot. Most of the time I just formulate the question I have in my head into something coherent and by that point I either have the solution, know what to search for or, in case it's not a question but a comment, I realize it's not worth saying.

</comment></comment></comment><comment><USER4> There are a number of comments that add up to "what steps will you take to ensure this does not happen again" - akin to a incident review. As speculation that's fine, as advice, I don't think it _should_ be listened to.

I am reminded of an long-in-the-tooth sysadmin of my acquaintance who logged in everywhere as root. His theory - "they are my boxes. I screw it up, I fix it." I eventually realised that typing sudo every time he touched a box was no defence against doing the wrong thing.

An awful lot of sites at 1.2m views would have outsourced the running and development of the whole thing - there are entreprenuers who say its not even worth our time to code up the MVP. I find this approach sensible from a business point of view, but still it does not sit right with me.

I am supposed to have a nice website with lots of good content to attract inbound marketing - so I tried getting someone on textbroker to write an article for me. It read like a High School essay - no life, no anime. And so I will probably write my own CMS and my own content.

And pg sits there and writes his site in his own language, with his own moderation tools. Apart from the hilarious idea he could find a ten person ruby shop to outsource to, its nice to see someone taking the time to play again. Its why I like to see jgc on here too.

I am not entirely sure those thoughts are joined up (I am procrasting like crazy) but if they come to mean anything its we are playing in pg's sandbox. If the sand leaks it's his sand, and the only company this is mission critical to is YC.

<comment><USER5> > I eventually realised that typing sudo every time he touched a box was no > defence against doing the wrong thing.

IIRC, sudo logs all commands to syslog. Which might come in handy. Yes, root commands will be logged by bash to .bash_history, but there are limits of # of commands lots, what happens if you are logged in multiple times into same account etc.

Anyway, that's why I like sudo.

<comment><USER6> Beyond the logging, which I love, I use it for the differentiation of states. I'm just a little more attentive when I type "sudo" before something.

At work, a relatively young engineer accidentally typed a command meant for a test database server into a production window. There was a big rush to restore from backups, and there was a small amount of data loss.

One thing that came out of the retrospective, requested by the engineer in question, was the _production hat_. Before you opened up a connection to a production machine, you had to put on the large pirate hat. You could only put it back when you had closed the connections. I didn't really need it, but it was a great way for people to learn the necessary caution.

It also ended up being a nice exclusive lock on futzing with production, and seeing it in use led to some good discussions that otherwise might not have happened. But the main thing was developing a strong differentiation of states in everybody's heads.

<comment><USER7> Another cheap solution is colorizing the prompt red for dangerous consoles and green for dev machines. This makes it very easy to notice when you selected the wrong terminal.

</comment></comment></comment></comment><comment><USER8> I'm not sure whether it's terrifying or relieving to realize that if all I dream of comes to pass and I achieve something akin to the legendary status of pg in the hacker community that I will still be susceptible to the inevitable facepalm moments that come with direct database access.

In any case I am thankful for the detailed explanation.

<comment><USER9> Some of the most spectacular airplane crashes are by the most experienced pilots.

If you've ever tried something new as a hobby you tend to be very careful. Once you gain confidence you take more chances and don't do what even a beginner might do.

<comment><USER10> There must be a rare personality type that never experiences this kind of overconfidence. Perhaps a less glamorous cousin to the Buddhist beginner's mind?

<comment><USER9> I find that if I am doing something "dangerous" (example might be using power tools) I have to say to my self "be careful this is dangerous" to avoid being on autopilot and making casual errors. Maybe a better example is the way you train yourself after you've picked up a box the wrong way and pull something to try to remember each and every time to watch your specific movements.

<comment><USER11> But at some point, you will become complacent, and it will taken a mistake to remind yourself again.

We've all done it. I shut down an NT4 production server because I was connected via remote desktop and clicked shutdown rather than log off. This was back in the day when there was no pop-up asking for reason you want to shut down and confirmation.

Luckily it was just our internal intranet server!

<comment><USER12> You were running NT4 Terminal Server Edition?

<comment><USER11> Yes, i think so. It was so long ago and it was my first programming role!

<comment><USER12> AFAIK that edition had logoff instead of shutdown on the Start menu for that reason. You can still access shutdown by hitting Ctrl-Alt-Del on the console or clicking Windows NT Security.

<comment><USER11> It was so long ago I have no idea. I defo vividly remember that I shut it down via the start menu, just one of many moments that stick out :)

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> Great postmortem and good lessons to learn here:

* Don't manually modify database without a well-tested procedure and another pair of eyes

* Don't leave persistent problems (e.g. memory problems) uninvestigated so that you miss new problems with similar symptoms

* Don't push new code to production while operational problem is ongoing (unless it addresses the operational problem)

I'm pretty sure I've repeated this exact same sequence before with similar results...

<comment><USER14> use CHECK constraints to prevent invalid data patterns when possible.

<comment><USER15> Sadly, even that isn't enough.

In our production database, I used CHECK constraints religiously. Worked great.

Then one day, I was no longer able to commit ANY transactions to a particular table, even completely innocuous ones.

The problem? The database _itself_ had violated its own CHECK constraint on a previous commit, but was _enforcing_ it on all subsequent commits, causing them to fail. Brilliant.

Moral: not even CHECK constraints will save you.

\----

P.s. This was a proprietary database, and when I reported the problem to the vendor (eventually, I figured out how to reproduce it), the vendor actually _refunded our (expensive) support contract_ rather than fix the bug -- they couldn't figure out how to fix it despite having a small bug report that reproduced the problem.

In the end, I actually had to remove the CHECK constraint altogether. :(

<comment><USER14> > The problem? The database itself had violated its own CHECK constraint on a > previous commit, but was enforcing it on all subsequent commits, causing > them to fail. Brilliant.

I've never heard of that, and unless you're using a really buggy, broken database, it should not be possible.

> P.s. This was a proprietary database, and when I reported the problem to the > vendor

well there you go. I think in practice, a simple CHECK constraint like the one we'd do here (literally, comment_id > parent_comment_id) is pretty easy to put one's faith into.

</comment></comment></comment></comment><comment><USER16> This should serve as a example template for how to accurately and transparently explain to users what went wrong. No deflecting blame, no useless platitudes.

Credit to PG, RTM and the rest of the team for keeping the sites uptime as high at it is.

<comment><USER17> "No deflecting blame"

Who were they going to blame?

<comment><USER6> He could have blamed the new server. Or whatever distracted him. Or the user, for being dumb. I've seen people do all of those.

Or he could have just dodged the blame entirely.

<comment><USER18> Some people are just freaking difficult to work with. I've worked with people that wouldn't accept responsibility even after every other possible cause was ruled out. I've even gotten this reply: "well, you must have been unlucky to get the faulty e-mail, because it seems to work most of the time". Yeah, because that's how programming works: cowboy coding and hoping for the best.

This guy actually called bugfixes "optimizations": "hey, X the feedback widget on the front page isn't working", "oh, yeah, I haven't worked on that because that's code that needs to be 'optimized', so it's low on my issues list". Ugh.

I've learned my lesson now. In fact, that's an incredible lesson for a startup founder: never, ever, hire someone who dodges a question on an interview. And the first time they avoid taking responsibility for something that was clearly their fault, fire them. The last thing you want is someone who'll blame everyone and anything else for their issues. It's a great way to kill morale and create rifts in a small team.

<comment><USER19> > And the first time they avoid taking responsibility for something that was > clearly their fault, fire them.

I guess there's still difficult for a lot of people to acknowledge their own mistakes, maybe because they're afraid of getting fired for that (acknowledging the mistake), which in the case of startups/small companies happens very rarely.

From my own experience of working at startups for my entire professional career as a programmer (7 and a half years) I can tell you that the first step when noticing you f.cked something up is to take immediate responsibility and then asking yourself "how can I/we fix this?" (you might need the help of other people to fix your mistake). After you've fixed the issue the question should be "how can we make so that this doesn't happen again?". That being solved I'd say nobody cares anymore whose fault was it to begin with, there's always other more important stuff to do.

I agree that maybe at larger companies this kind of thing might happen exactly the opposite way, i.e. you can get fired for making a mistake and nobody really cares to fix other people's stuff, because their next paycheck/financial well-being does not depend on that (or so they think).

<comment><USER18> Exactly! When everyone is on the same boat the priority is _fixing_ the stuff, then wondering whether attributing responsibility is important (most of the time it isn't. Who cares who fucked up the e-mail template, as long as it's fixed next time it runs.)

I think in this guy's case the causes for his reluctance (or rather, incapability) to accept his own mistakes had much deeper roots. He was literally the most self-centered person I've ever met, to the point that he wouldn't accept anyone's opinion on anything. He went out of his way to find doctors that'd go with his suggestions and run all kinds of tests on him to determine why he had a blood pressure problem, when he was clearly _way_ overweight and had the unhealthiest diet I've ever seen. He'd dress up in shorts and t-shirts during the worst days of winter, and then take a niacin pill to force a capillary rush so his hands wouldn't feel cold (?)

Basically, he just thought the world had to bend to his will. Why use common sense, when you can just say "fuck it" and find a workaround that fits your mindset. Of course you can't expect someone like that to 'accept' his own shortcomings.

The scariest part of all this is he tried, for a while, to become a cop. Yep, imagine that: a 240lbs armed prick, completely unable to reason, forcing his way on everyone. I shudder at the thought.

<comment><USER20> > imagine that: a 240lbs armed prick, completely unable to reason, forcing his > way on everyone. I shudder at the thought.

Where are you from where that's something you have to _imagine_ , because I want to move there.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> "But then I decided to just fix it for him by doing some surgery in the repl."

I've always found it's a good idea to not deviate. Whether it be running, parking or anything else once you deviate from some regular behavior you run into potential problems that you hadn't anticipated.

"For some reason I didn't check the comments after the surgery to see if they were in the right place. "

More or less my point. If this wasn't a deviation from normal behavior you would have "checked the comments after the surgery" because it would have either become habit or the shear number of times you tried a fix resulting in an error would have made that more likely to occur.

<comment><USER21> > I've always found it's a good idea to not deviate.

Aren't you assuming "surgery in repl" is a deviation? What if it's normal course of action for him?

> More or less my point. If this wasn't a deviation from normal behavior you > would have "checked the comments after the surgery" because it would have > either become habit or the shear number of times you tried a fix resulting > in an error would have made that more likely to occur.

How about the opposite scenario? He has done it so many times with desired results, that he didn't bother checking?

<comment><USER22> > Aren't you assuming "surgery in repl" is a deviation? What if it's normal > course of action for him?

This is a big difference between engineering and hacking. An engineer would never _regularly_ do something so dangerous.

But I suspect pg isn't an engineer when he works on HN, I suspect he is a hacker, and just does whatever he wants to, whenever he wants. Which is his prerogative.

</comment></comment></comment><comment><USER23> Why do "self posts" like this show up in the same light gray as posts with negative vote counts? My eyes aren't great and I find it hard to read

<comment><USER24> The rationale for this is that if you need to post a long text post, it should be in the form of a blog post instead. I agree with you that it's not really adapted for a meta post.

</comment></comment><comment><USER21> Disclaimer: Hindsight is 20/20, and stuff.

If reverting code didn't fix it, reverting server didn't fix it, incorrect data is the most likely culprit(I am not claiming this should have outright occurred to you; just thinking out loud). I take it you introduced non terminating recursion by making a thread its own parent, and you made the change on disk.

But this analysis is the last thing that comes to mind when you already have introduced 2 new variables the same day - new code, new server. And an old, recurring variable(GCing too much) is in play as well.

</comment><comment><USER25> So what do you do to avoid this in the future? Do you stop doing surgery in the repl, or do you do the surgery with functions that check for cycles from now on?

<comment><USER22> > So what do you do to avoid this in the future?

It's HN... there's no SLA, there's no postmortems, there's no doing things better in the future. pg just runs this site out of the good of his heart, we should be lucky the volunteers run it for us at all.

<comment><USER26> > pg just runs this site out of the good of his heart

Hilarious. I would have believed you if you appended "and his wallet"

<comment><USER22> One core. One HD. Bandwidth is trivial with no images. How much do you think this site costs to run?

<comment><USER27> He means that PG runs the sight because it makes business sense, not out of charity.

</comment></comment></comment></comment></comment><comment><USER28> "We'll do it live!"

<comment><USER29> "Hey, Hold my beer and watch this!"

</comment></comment><comment><USER30> This is a particularly endearing piece of "hacker news". It's so easy to relate to.

</comment><comment><USER31> Are you saying you manually modify the database? Like, shifting around things by id instead of just making admin buttons next to posts?

<comment><USER21> HN runs on plain files. He wasn't modifying database, but calling functions(I believe) in the repl to change the parent id of the thread.

But that apart, even if there were an admin button to change the parent id of a thread, he would still have made the same mistake.

Unless the code in question was checking for loops. In that case, repl would have worked the same.

<comment><USER31> I sort of meant that you shouldn't modify things like that directly. Be it a filesystem, database, or any other place that makes it possible to mess things up to bring a rather strong server down.

<comment><USER21> I see where you are coming from. But I am saying this didn't happen because he did things live. This happened because he entered incorrect id making a thread its own parent(or grandparent; doesn't matter).

This is the kind of mistake one would make even if you were writing proper migrations. He was doing things live isn't an issue; neither is an incorrect id. The issue is the code doesn't check for loops.

<comment><USER32> I am but an egg, I have two questions.

One, if the data were held in a database, should a change like this be captured in the database logs? I am seeing more and more situations where I want these, I notice that they are by default turned off for mysql and wonder if this reflects a de facto judgment that logging slows performance more than is usually worthwhile.

Two, if the data were kept in a database, wouldn't something like this be prevented by a constraint preventing a comment from making itself an ancestor? But I suppose there is a slight performance hit in checking such constraints, and the case arises so rarely that this hit isn't generally worthwhile.

<comment><USER33> Databases, at least the SQL kind, really aren't good at dealing with hierarchical data, and I don't know how you'd even begin to express that kind of constraint. I don't think a traditional database is the answer here. (If it were me, once I'd done it more than twice I'd write a "move thread" admin tool in the UI, and after I screwed it up like this I'd have a place to add such a check to).

<comment><USER34> If you were using some kind of representation for Nested Sets -- left-to-right depth-first numbering, or a human-readable id.id.id chain -- then it's really easy to write a constraint for that: parent left < myleft, right > myright, or dotted_id.split('.').filter{|first, rest| return false if rest.contains first} (yeah, yeah, that second pseudocode would be unrealistically PITA for some DBs).

More generally:

I'm not a big SQL wonk anymore, but I find a lot of people have the intuition that relational databases are ill-suited for trees.

An intuition that is much closer to the truth is that almost all databases can handle trees pretty well, because there's still an unambiguous concept of ordering and containment, and you can usually arrange things so as to do range/ancestor/inclusion queries efficiently.

It's graphs with loops/without unambiguous concept of ordering/containment that are really hard.

<comment><USER35> Found this: The excellent Postgres documentation includes an SQL graph search with two different ways of graph cycle checking, here: <http://www.postgresql.org/docs/9.0/static/queries-with.html>

One way involves accumulating an array of nodes already visited as the tree gets walked, checking each node as-visited for membership in the array-to- date.

The other method, a bit more of a hack, is just adding a LIMIT clause.

I think the 'WITH' clause is a great addition to the SQL standard, very much worth the learning the weirdness of its syntax and its optional 'RECURSIVE' term (which, as the Postgres documentation points out, isn't really recursion, it's iteration).

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER36> Do you have munin monitoring on the production HN server?

That would really make situations like this easier to debug. First, it can pinpoint exactly when something started happening, which in this situation might have helped you realize the problem was caused by your change. Secondly, in this specific situation it probably would have been easier to differentiate a situation where you are running low on memory vs this completely different situation.

As somebody who spent a lot of time professionally debugging large software systems when they were misbehaving (as a Google SRE), I can tell you that looking at graphs of many key metrics (disk IO, CPU, memory, then application specific things) was always the place to start when debugging a situation, because you can learn so many things right away. When did it start? Was it a slow buildup or an immediate thing? What is the general problem (Memory?, Disk IO?, CPU?, none of the above?)? Has a similar pattern happened in the past?

Then you can start to get fancy and plot things like "messages/minute" or something and then it becomes easy to see when issues are affecting the site performance and when they aren't.

<comment><USER10> That and something like the Smalltalk Change Log would have made this a no- brainer debug. (Yes, every REPL action in Smalltalk got logged by the same mechanism that logged every code change.) Such mechanisms aren't trivial, but they're not rocket science either, and they have tremendous ROI.

</comment></comment><comment><USER37> I wonder what exactly did distract you :) When I do surgery on a production server, I triple-check making sure everything works properly.

I have two assumptions: 1. HN has a low priority in the overall scheme of things, 2. Self-confidence overflow :)

</comment><comment><USER38> Happens to a lot of us. Great reason to always write tested cleanup scripts for this stuff instead of editing directly on the server. The only time I brought down my product last year was from a similar screwup, I was removing users by hand and somehow managed to end up with a 0 in my list of user ids, thus deleting the anonymous user, and causing havoc to my server, which took a long time to track down.

</comment><comment><USER39> Thanks for the detailed explanation.

It sounds like everything was done to fix the problem _except_ try to figure out what the problem actually was. Why not use tools to see what the program is doing, form a hypothesis, gather data to confirm or reject the hypothesis, repeat until cause found, and then take corrective action that by this point you have high confidence will work?

I realize HN is more of a side project than a production service, but the goal is the same in both cases: to restore service quickly so you can move on to other things. It feels like a more rigorous approach would allow restoring service much faster than randomly guessing about what could be wrong and applying (costly) corrective action to see if it helps.

Besides that, in many cases (including this one), you cannot randomly guess the appropriate corrective action without finding the root cause.

</comment><comment><USER5> I use assertions to protect against things like this.

I liberally sprinkle my code with assertions (CS theory calls them pre- conditions and post-conditions, iirc) to crash early if the system is an invalid state.

One my pet peeves is that few programmers seem to love assertions like I do. Would love to see to comments on this.

<comment><USER40> What assertion would you have used in this case? For every comment you'd have to iterate through all it's parents to check if there is a cycle, which seems pretty inefficient to do for something that should never happen (there are other ways that you could check for this problem as you go, but the only other ways that I can think of require holding extra state just in order to perform the assertion).

I'm for assertions when they are simple and don't cost much (especially during development), but it's not feasible to check every condition that should not happen.

<comment><USER41> You could assert a limit on depth, perhaps. Then the cycle would still exist but after X number of comments, the rendering ends.

<comment><USER40> This is a reasonable solution. While it will (almost) never provide the correct result (it might print out a cycle of comments until X is reached, or it might cut off a very long but legitimate comment thread), it would provide a reasonable guarantee on this sort of problem not generating infinite pages.

<comment><USER41> At the risk of being accused of flame-baiting, I'd say it's the engineering solution rather than the mathematical one.. ;-)

For some reason I tend to be a fan of the "stick it in a secure box" rather than "get it right in the first place" approach..

</comment></comment></comment></comment></comment><comment><USER42> Hacking code in the repl without testing the new behavior. We all did that. Don't lie. Once I wanted to quick fix a "gmail.ca" to "gmail.com", which I did.. but to all the users instead of just the one mistaken. Fortunately I realized by mistake really fast ;-)

</comment><comment><USER43> The pink sombrero could have saved HN: <http://www.bnj.com/cowboy-coding-pink- sombrero/>

</comment><comment><USER44> Appreciating the details.

"Hacker News was down all last night."

With the internet there is no "last night" ;-) Europe - and more so Asia I assume - had to live for many working hours without HN.

<comment><USER45> Yeah, I was more productive yesterday. But I did read google cached versions of HN.

</comment></comment><comment><USER46> PG, quick question: Did this impact the server hosting the YC Summer 2013 applications?

When I tried to edit mine, it simply said "Thanks, scotthtaylor"

<comment><USER46> Working again now.

</comment></comment><comment><USER47> >>I caused an item to be its own grandchild.

Please forgive me. I know you folks tend to hate jokes on here. Don't waste your time if you're immune to corny humor. "I'm My Own Grandpa- Ray Stevens" ( with family tree diagram) <http://www.youtube.com/watch?v=eYlJH81dSiw>

</comment><comment><USER48> Great story! Yup, this kind of thing happens. For some reason it reminded me of something that happened to me as a newbie engineer. It was really funny a week later.

I was troubleshooting an intermittent problem in a piece of equipment. It had several boards full of mostly LS TTL logic chips (yes, them chips). It was the kind of problem that only happened once every other day or two. Nobody knew. So, I had all kinds of instruments attached to this thing and was watching it like a hawk waiting for a failure. It had probes attached to every point in the circuit where I suspected I could see something and learn about the source of the problem. I also tested for thermal issues with heat guns and freeze sprays, familiar troubleshooting techniques to anyone who's done this kind of thing.

Anyhow, every so often the thing would go nuts. The three scopes I had connected to it showed things I simply didn't understand. I'd analyze but couldn't make any sense out of it. Still, again, every so many days it would happen again. Changed power supplies and the usual suspects. No difference.

Well, finally, two weeks later, the other engineers in the office took pity on me and told me what was going on: They had connected a VARIAC to the power strip I was using to power the UUT (unit under test). The scopes and other test instruments remained on clean power. Every so often they'd reach into this drawer where the VARIAC was hidden and lower my power strip's voltage just enough for the power supplies to fall out of regulation and everything start beeping and sputtering. Those friggin SOB's. They had me going for days! I was pissed beyond recognition. Of course, after a while I was laughing my ass off alongside them. Good joke. Cruel, but good.

My revenge: A CO2 fire extinguisher rigged to go off into his crotch when my buddy sat down to work.

Fun place to work. We did this kind of stuff all the time. Today I'd be afraid of getting sued. People have really thin skins these days.

</comment><comment> DanI-S n.b. that this is why time travel is a _terrible_ idea.

<comment><USER49> The grandfather paradox is the best solution to the halting problem.

</comment></comment><comment><USER50> Funny to see that this happens to everyone. A week ago, while testing some stuff to locate a low-importance bug, I erased the whole user database. Fortunately we have a good restore so the problem was solved in a few minutes, but still, cold sweat here ...

</comment><comment><USER51> Isn't it curious that the comet incident over Russia happened so close to the pass of DA 14. In the intro to the book:

<http://ruby.bastardsbook.com/about/#why>

is the note about surgical instruments left inside. It seems just like a coincidence that this happened so close to the switch to the new server, but I wonder if it's something deeper in the subconscious mind; the change to the new server is quite a big change (I know I feel that way when I have purchased a new computer (it feels different - even if it's running the same linux as before)) and could have upset the normal checks one has in place when tweaking things.

</comment><comment> cool-RR Great debugging story!

I guess the lesson is to have code that alerts you about comment loops without going into an infinite loop.

Also another lesson would be to figure out a way to have better clarity into which requests are causing a timeout on the server.

</comment><comment><USER52> I'm curious how was RTM able to notice that the problem seemed related to a specific item id? It would be great if he might write a short blurb similar to yours. Which also makes me wonder, why does RTM not write much?

</comment><comment><USER53> I absolutely love post-mortems like this. It clearly identified that there was a problem, what the author tried to do to fix it, and if it was successful. Even if it ends with the author not knowing too much about the solution that was used, it's still so interesting to see the workflows and be able to derive something from it.

It's also why I like to read pg's articles so much. They're so in-depth and detailed and it doesn't feel you left thinking something was left out for the sake of being hidden.

</comment><comment><USER54> I fall into a similar misdirected-focus trap, but mine is simpler: I waste an embarrassing amount of time in editing the wrong damned file. After a sequence of small tweaks that yield no change in the results, I make a huge change and see nothing, and then realize that I've done it yet again. I need to write a vim macro that blanks the screen every few minutes, displaying the message "are you SURE this is the right file?"

</comment><comment><USER55> > created a loop in the comment tree; I caused an item to be its own > grandchild.

Ah, the online forum equivalent of going back in time to kill your grandfather.

</comment><comment><USER56> It's nice to know that even the mightiest of us can still make mistakes. Thanks for being willing to admit mistakes so the rest of us can learn.

</comment><comment><USER57> Thanks for fixing it.

Have you considered avoiding dipping into the repl to do these kind of fixes? You don't owe any of us any sort of uptime guarantee, and you're a much better programmer than I, but it strikes me as odd that you would hack against the live server instead of create some tool that would make it so you couldn't take down the whole site when making this kind of fix...

</comment><comment><USER58> Thank goodness it's back. I lost the my meaning of existence for the entire day. I don't know where my yesterday went. I'm ok now. :)

</comment><comment><USER59> It's never what you think it is. One time, I had a memory leak in a Rails app that took me TWO WEEKS to find. In the end, it came down to me putting a line of config code in the wrong section of the config file, which for some reason created a recursive loop and caused my servers to crash about once every 30 minutes. #weak

</comment><comment><USER60> Whoa, what an unfortunate coincidence. This whole bug would be so much easier to find, if it weren't for the new server.

<comment><USER33> The bugs that actually hit production are always like this - a confluence of three or so factors - because if it were simpler you'd have caught it earlier.

(Though I have to say, upgrading the code at the same time as you're restarting to fix a problem is really a rookie mistake. It's incredibly tempting because it saves so much time, but if you do it you _will_ get it wrong sooner or later. One of the hardest skills in programming is acquiring that zen that you need to wait in a state of readiness for the effects of your first change to make themselves apparent, rather than changing something else)

</comment></comment><comment><USER61> That's funny -- I work in genealogy software, and loops ("being your own grandpa") happen all the time, due to data entry errors. To avoid infinite recursion, we always keep track of what records we've processed already, check whether "I've been there before", and bail out if the answer is affirmative.

</comment><comment><USER62> You are honest and I respect that. I'm sure many companies try to play off their downtime as something far more sophisticated when in fact, it was something too embarrassing to admit. I've certainly had my fair share of embarrassingly stupid mistakes that resulted in downtime.

</comment><comment><USER63> Cheers for that pg - now I have to explain to my boss why I was _actually_ productive yesterday.

</comment><comment><USER64> My pet peeve: You made an arbitrary change while debugging a problem. NOW YOU HAVE N^2 PROBLEMS!

</comment><comment><USER65> That is why some organizations don't allow adhoc data fixes to be run in production. Best practice is to backup the database, run the fix against the backup, test the fix against the backup, and all being well run the fix against production.

</comment><comment> T-zex Thank you for the honest explanation. This is not so easy especially for a famous person.

</comment><comment><USER66> You should probably make your code robust to this sort of data corruption in the future.

</comment><comment><USER67> "I don't know if Nick succeeded, because in the middle of this I gave up and went to bed." - Not a good example to your holding companies :) What would happend if they all went to bed when something goes wrong :) Just kidding.

</comment><comment><USER68> Just about anyone that has programmed for any length of time has done something like this. It is one of those "fixes" that after it's actually fixed you try to never think of it again. Good to know PG is mortal. :-)

</comment><comment><USER69> Related question: what is the timing for the 'Reply' link to show up? I might be fantasizing but sometimes it takes 5, sometimes 10 minutes to appear, leading people to reply as a sibling instead.

<comment><USER70> Deeply nested comments tend to be hot, and so the reply link takes a while to show up to try to give people some time to think about what they're going to say.

<comment><USER69> Interesting. So it slows down discussion as it progresses, until it either stops, is forgotten or becomes a series of long essays.

</comment></comment></comment><comment><USER71> Seems related: [http://meta.stackoverflow.com/questions/66377/what-is-the- xy...](http://meta.stackoverflow.com/questions/66377/what-is-the-xy-problem)

</comment><comment><USER72> When hacker news is down, I finally lifted my head and realized that there is life beyond the screen on my phone.

Now that it's back. I realized that it's finally time to create an account :)

</comment><comment><USER73> It seems that in your new server and also latest pushed code, I can't do `like` anything. Honestly it's not a new bug and I got used to that, don't think about that.

</comment><comment><USER74> That explains something weird I saw.

If I went to Google's cached copy, I could see threads, and then click on them. But the front page was down. But I could see individual threads.

Very confusing.

</comment><comment><USER75> Did you add code that detects very deep nesting levels (e.g. depth more than 100) and throws meaningful exception to help developers to diagnose the problem?

</comment><comment><USER76> it's a good job it's your site, this type of thing is often what gets someone fired in a company. Modifying (meddling!) the production system directly.

<comment><USER77> I disagree. Very few companies would think negatively of an engineer if they made such a mistake on a non-essential, non-revenue-generating fun/research project.

How many dollars did YC lose because of the outage? None. (Maybe they saved a few on bandwidth!)

I also predict that exactly zero startups will say, "Man... I'm not going to take seed money from _those_ guys! They had discussion forum downtime."

<comment><USER76> They could save even more if they shut it down! That's a ridiculous thing to say. We could all save money that way.

You've obviously never worked in a for profit corporation. in such there are policies and practices put in place to prevent just this kind of newbie mistake. You never modify the live database directly. Never ever. Whether it's a bottom line property or not.

I didn't say it would negatively impact YC's business. It might make them look incompetent, but these things happen, but people don't approach YC for their website savvy, they go there for the money and the connections. Most of the VC firms i've EIR'd at have much worse IT than hn. Their sites are barely usable. It seems to just go with the territory.

Let's not be so defensive, PG can do what he likes with his site, including take it down whenever he feels like saving bandwidth. But in the real world these kinds of things get real people on a fast track to their exit interview.

<comment><USER77> "You've obviously never worked in a for profit corporation"

Wow, really? I don't think that attitude is warranted at all.

At any company (for-profit or otherwise) there is a finite amount of time and money -- and surely we can agree that solid development/deployment practices carry an upfront time/money cost, can't we?

In an ideal world, all projects would have continuous build processes, automated tests, and management tools extensive enough to render live database surgery unnecessary.

Perhaps you've worked at companies so flush with cash that every single line of code, research project or otherwise, has gone through rigorous development/testing/deployment practices. If so, I'm jealous. I've always worked at companies that had to be choosey about how they spend their resources.

</comment></comment></comment></comment><comment><USER78> Don't worry I just figured out the totally bone-headed programming mistake I made at noon today. Time is a good mediator between skill and stress.

</comment><comment><USER79> Much appreciated, pg. I knew that the "10 minutes of downtime" would not occur (fair enough, this was not related to the server upgrade).

</comment><comment><USER80> Ok, I'll just say it: that's just plain dumb. It's a rare case, but a simple condition would have checked and prevented this. :)

</comment><comment><USER81> Did you hear about the tortious and the hare?

</comment><comment><USER82> Everyone fat-fingers a database at some point... Then you build interfaces so that you can't make the same mistake.

</comment><comment><USER83> Why does PG maintain the website himself? I would think he would have many better things to do with his time.

</comment><comment><USER84> That's funny because one of the top posts in progit yesterday was about the Hare and Tortoise algorithm

</comment><comment><USER85> Thanks for the explanation pg. As you said in the original thread, "you know how these things go..."

</comment><comment><USER86> The nice thing about surgery with a computer program on a server is that death is not permanent.

</comment><comment><USER87> It makes me feel good knowing better programmers than me go through the same issues I face. :)

</comment><comment><USER88> I think this demonstrates how many people browse the /threads?id=pg page (myself included)

</comment><comment><USER89> TIL there are still large web sites out there that do not have staging environment.

</comment><comment><USER90> I appreciate (if not love) the fact that you bugfix and server-change yourself.

True hacker spirit.

</comment><comment><USER91> _Sorry about that._

No worries.

So, are we back on the new server? Or was this too much for one transition :)

</comment><comment><USER46> Normality has returned :-)

</comment><comment><USER92> I was almost sure it was Anonymous! ... Are you in Anonymous, pg? :D

</comment><comment><USER93> Same problem in Iran, I couldn't access to HN all last day.

</comment><comment><USER94> I guess it's time for NewRelic to add an Arc agent.

</comment><comment><USER95> That sort of thing is fine for a startup in it's first year or two of life, but HN has been around for a while now ... surely you must have some sort of process by now?

</comment><comment><USER96> So, uh, still fixing stuff in production?

</comment><comment><USER97> The cobbler's children have no shoes :-)

</comment><comment><USER98> pg,

Just wondering as to why HN isn't hosted in the cloud? (e.g. on AWS, Rackspace etc.). How do you backup all the data?

<comment><USER99> Because that would cost more?

I don't really know what the benefit of cloud hosting would be in this case.

</comment></comment><comment><USER100> I appreciate pg's frankness here.

</comment><comment><USER101> i hope that user wasn't me. i was editing a typo in comment right when it happened

</comment><comment><USER102> I think we have a new king!

Awesome explanation.

</comment><comment><USER103> "I am my own grandpa"

</comment><comment><USER104> Even Homer nods!

</comment><comment><USER105> >On a comment thread, a new user had posted some replies as siblings instead of children. I posted a comment explaining how HN worked. But then I decided to just fix it for him by doing some surgery in the repl.

No good deed goes unpunished!

People sometimes reply as sibling because they too impatient wait for the built-in delay on child comments.

Thanks for keeping the experiment going.

</comment><comment><USER106> If the problem existed before the code update, why would you assume it was the code update that caused the problem?

</comment><comment><USER107> After breaking many things myself due to similar, seemingly miniscule edits, I have implemented an ABC routine: Always Be Checking. Even if it was "just" something like moving a piece of code or something equally tiny, I always check after the fix.

So far, it has been working great.

</comment><comment><USER108> Are you sure that comment's name wasn't Phillip J Fry?

<comment><USER108> Apologies for the trivial comment</comment></comment>


A Nixon Deepfake, a 'Moon Disaster' Speech and an Information Ecosystem at Risk - <AUTHOR> https://www.scientificamerican.com/article/a-nixon-deepfake-a-moon-disaster-speech-and-an-information-ecosystem-at-risk/ <comment><USER2> Currently being discussed in another thread:

[https://news.ycombinator.com/item?id=23896996](https://news.ycombinator.com/item?id=23896996)

<comment><USER3> Comments moved thither. Thanks!</comment></comment>


Google limiting and generalizing political ad targeting around the world - <AUTHOR> https://9to5google.com/2019/11/20/google-political-ad-targeting/ <comment><USER2> Why is this a good thing? In the U.S., why should Democratic candidates for the presidential nomination waste their ads on Republicans who will not vote in the Democratic primaries?

"The company's advertising networks are limiting election ads audience targeting to age, gender, and general location (postal code level). This includes Search, YouTube, and display ads that appear on third-party sites.

Previously, Google's basic political targeting in the US allowed for ads to be served based on public voter records and general political affiliations (left- leaning, right-leaning, and independent)."</comment>


Ask YC: Entrepreneurs vs Builders - <AUTHOR>

This is an ill-formed question/debate topic that I've wanted to broach for a while.  Sorry if it's not as clear as could be hoped, but it's sort of a vague notion.
One of the things I've liked about PG and his writing is that I get a sense of a hacker figuring out enough business stuff to make a go of it, and consequently flipping the company: as a hacker, he was never going to want to stay on the business side anyway.  PG strikes me as a "builder" - someone with lots of technical ability who goes out and creates things.  Incidentally, I also see myself as being more along those lines than a "business guy".
"Entrepreneurs" on the other hand, are those "wheeler dealer" type of guys who are really good at putting together various people and other aspects of a business.  They are kind of technical, but not the guys who go out and build something really cool themselves.  They're the classic business starters, in my opinion.
So - what do you guys think of "builders" vs "entrepreneurs" in this day and age? <comment><USER3> I think what you are proposing is a false choice but I understand why you want to frame it this way.

Having been an entrepreneur for the last fifteen years, I have come to learn that my __job __description as an entrepreneur is simple, "if I could __describe __it, it ain't my job".

In other words, in the beginning of the startup, you have to do whatever it takes (be an engineer, be a salesman, be a negotiator, etc.) but once the company gets off the ground, then you need to give that responsibility to someone else. But by then, there will be other tasks that are not well defined that require your attention. And the cycle continues until the company is no longer a startup and the entrepreneur has lost his/her place in the company (or he/she matures into a manager).

At the end of the day, an entrepreneur's job is to connect the dots. It is a deceptively simple description of what we do.

<comment><USER4> Historically the distinction was far from false: it's Watt vs Boulton.

Perhaps it's getting falser as techies increasingly don't need business guys as cofounders.

<comment><USER5> "Perhaps it's getting falser as techies increasingly don't need business guys as cofounders."

The reason the barriers to starting a business are falling is because the tech side is getting easier and cheaper. The business skills one needs remain unchanged.

<comment><USER3> I agree. My own experience is that entrepreneurs tend to greatly underestimate the difficulty of selling. Particularly on this forum, we talk a lot about "making something people want" as if that's all it takes for a startup to succeed but we don't talk enough about "making customers pay for what they receive". My own experience is limited to selling to enterprise customers. In that case, the necessary condition is to provide a solution that puts out a fire that is burning someone's butt. But it is far from sufficient. In the end, it all comes down to convincing someone to risk their career and the financial future of their family on an unproven product from an unproven startup. Having a killer product is only the first step.

</comment></comment></comment></comment><comment><USER6> I think it totally depends on the business, and the core problems that need to be solved for the business to win. Some businesses require only technical genius. Some really just need an innovative UI. Some need dealmakers.

A lot of builders attack problems whose core problems aren't technical... Which often leads to failure.

Either way, in the earliest stages, specialization is oftentimes a really bad thing. If you can't sell your product to co-founders, employees, partners, and customers, then you might be too specialized to be a successful entrepreneur.

Entrepreneurs (whether they are builders or not) get shit done. PG did sales. When he was starting up, he aggressively recruited great team members. Great entrepreneurs don't silo themselves... They stack rank the most important problems they should tackle (whether they are building-related or not) and then they try to destroy 'em with whatever means necessary.

So, in short: Make sure your teams' skills match the core issues of the product you're attacking, but know that every team member is going to have to get out of their comfort zone and attack stuff that needs attacking.

</comment><comment><USER7> I think I have a slightly different spin on the difference. I tend to see, rightly or wrongly, a Builder as someone who is going to see a job through to the finish - a details person, in essence. A Hacker, in YC parlance.

I see Entrepreneurs as those that create the ideas, or see the vision of how an idea could be used, implemented (on a broad level), or sold. I feel an Entrepreneur's skillset lends itself to creating the team, inspiring the vision and making the deals necessary to establish the company as a viable entity.

I also tend to see Entrepreneurs as quickly bored (perhaps that is just how I see myself) and eager to move onto the next thing.

I guess that's why I feel a startup generally needs two co-founders - <USER2> business person and a techie person. Ideally, they'll each have a degree of appreciation for the other's domain.

</comment><comment><USER8> I believe that there's an inverse relationship between "building" and "wheeling and dealing". The more and better you build, the less you need to wheel and deal.

I've always noticed that commodity type products needed great promotion people. Examples: insurance, real estate, some industrial products.

I think this happens less with great products that address a real need and have captured their place in the market. You don't see many telemarketers for Apple, Volvo, or Google.

I focus on building so I won't have to wheel and deal as much. Might not be a bad idea for most people here, too.

<comment><AUTHOR> > The more and better you build, the less you need to wheel and deal.

I think that that is actually a common mistake for 'builder' type people. "I'll make something that kicks ass, and everything else will sort of work itself out".

<comment><USER8> "the less you need to wheel and deal" != "everything else will sort of work itself out"

Big difference

<comment><AUTHOR> Ok, I think I used 'wheeler dealer' while thinking of something more like what Denny describes. Not a shyster, but getting out there, promoting your stuff, doing deals, meeting people, and otherwise getting people on board. And I think that you absolutely have to do that, even if you've built something fantastic. Where would Woz be without Jobs? You need someone who can get out there and schmooze/sell/hype/etc.. your product.

</comment></comment></comment></comment><comment><USER9> I think that's a really good question. There are (at least) two distinctly different skill sets or personality types essential for a successful entrepreneurial business - one is the thinking-of-new-ideas-and-making-them- work track and the other is the bookkeeping-and-keeping-up-with-routine track. I'm not very good at the second; switching back and forth between coding and bookkeeping makes me want to sit in a corner of a dark room and rock back and forth slowly while reciting Kipling. Some people, though, are really good at the second. I know a few people who manage both and I'm rather in awe of them.

Perhaps there's a third type, too - the guy who goes out and sells the idea, like Jobs to Woz.

How do you solve this problem? What am I doing wrong?

</comment><comment><USER10> Entrepreneurs are defined by their characteristics: Willing to take risks, able and willing to learn and execute on multiple areas of business starting, and most of all, adaptable: able to change course based on what he or she sees in the market.

All entrepreneurs are builders. They build both the idea and the business. What separates them from builders is their ability to adapt and their willingness to branch out into a lot of different areas

</comment><comment><USER11> Would you classify Steve jobs as a "builder" or an "entrepreneur"? i think for any would be successful startup founder, you have to swing both ways. just my 2 cents.

<comment><USER12> Jobs is not the "builder" in any technical sense of the word. But he's a brilliant entrepreneur nonetheless.

<comment><USER11> Steve Jobs and Steve Wozniak were the main brains behind the Apple computer although Steve Wozniak was more nerdy, Steve Jobs also had his input. He is seen as a business or idea man today because of his charisma and flamboyance.

</comment></comment></comment><comment><USER13> People who can run the day-to-day operations, dealing with many dept. (sales, marketing, technology, HR etc.) may or may not be an entrepreneur because my understanding is entrepreneurs are builders because they create something from scratch.

Many people start their career with 9 to 5 job and keep getting promoted to the CEO level, but they may not have ability and courage (at least in the beginning of their career) to build a new business right from the scratch.

While entrepreneurs/builders are the people who start and build something (or help others to build something) from nothing.

Serial entrepreneurs (and builders) are always on the move - from one project to another as they are not interested in doing the same thing over and over again on daily basis - they get excited to build something from scratch and once it is built they find the right person to run it (or they help others to build it and move on to the next project).

Serial entrepreneurs (and builders) to me are like King-makers, they are not interested in becoming a king them self and run daily operations. They are like a Guru who has many disciples (kings) who achieve success with his/her skills and knowledge. It is well known saying that a Guru is equal to a 1000 libraries - they know which book is for whom, and when and how to use it.

<comment><AUTHOR> > entrepreneurs are builders because they create something from scratch.

Yes, the two types of people aren't _that_ far apart, but the entrepreneur is the guy that can tie things together and make things happen, whereas the builder is the one who creates a beautiful, well done product - or maybe not even that, just a brilliant idea that no one has ever done before. Perhaps "pure" builders aren't suited for startups - they're the brilliant type of people who just don't want to be bothered with organizational or marketing details, and would be happy in the R&D department of a company big enough to give them all the resources they need to create really interesting things. At least that's my mental image...

Some hackers who strike me as not wanting to deal with the business side, and have still prospered might include people like Andrew Tridgell (who is a seriously smart individual), Linus Torvalds, Guido van Rossum, Alan Cox, and so on. Not that they don't have some good organizational skills and people skills too, but they are not the kind of people I would describe as entrepreneurs.</comment></comment>


YouTube Demonetizes Videos with Titles Containing LGBTQ-Related Words - <AUTHOR> https://www.youtube.com/watch?v=ll8zGaWhofU <comment><USER2> For those who don't feel like watching the video (it's long and a bit dramatic), the group has put together a spreadsheet for viewing. Note that this is ALL banned words, not just the ones this author has issues with, so the content is decidedly NSFW. Proceed with caution.

[https://docs.google.com/spreadsheets/d/1ozg1Cnm6SdtM4M5rATkA...](https://docs.google.com/spreadsheets/d/1ozg1Cnm6SdtM4M5rATkANAi07xAzYWaKL7HKxyvoHzk/edit#gid=1380702445)

Honestly most of that list looks like a standard blacklist, but there are a few items in the list that have become more socially acceptable recently. But I'll refrain from calling out anything specific; form your own conclusions.

</comment><comment><USER3> When wrongthink is defined as anything that isnt rightthink, rightthink can change from moment to moment, and the consequences of deviationism are unthinakble: we'll have the perfect consumer culture. NoThink. (TM) all rights reserved and ask your doctor for a prescription.

</comment><comment><USER4> This is really awful behavior by Google. We need to eject Google from LGBTQ+ organizations so long as Google chooses to actively hurt us.

Don't let Google wrap itself in our flags while stabbing us in the back!

</comment><comment><USER5> The internet has become too centralized.

</comment><comment><AUTHOR> This just fell off from the main page, and is nowhere to be seen in the second or third or anywhere else but "new". Is this normal behavior or did it get soft-flagged or something?</comment>


Show HN: Share chess positions and moves - <AUTHOR> http://webchess.cygx.de/

<comment><AUTHOR> Something I hacked together during the weekend. The purpose is to provide a mechanism to easily link to arbitrary positions like

        http://webchess-cygx.rhcloud.com/?n_rb5_p3_p_p_p7_R5_K8_p3_p_PN_2P_R3_N6_k-w--

As a bonus, you can see what others have done in a given situation.

In principle, you could use it to play cooperatively against any- (or rather every-) one.</comment>


If you're a perfectionist - <AUTHOR> https://medium.com/@shkesar/if-you-re-a-perfectionist-50b233bef61b

<comment><USER2> The problem is knowing when enough is enough. When doing work for a client, asking myself "would a client be happy to see this?" helps. I produce good work but do so in good time. This might work well with side-projects, where there's a greater danger in over-doing it.</comment>


Boris Mann's response to BootupLabs portfolio reduction outcry - <AUTHOR> http://bmannconsulting.com/2843/personal/keep-calm-and-carry

<comment><USER2> Quoth Boris Mann in the linked blog post: " _I read a number of comments / response to me directly that suggested that we should have been more apologetic. We leaned towards the factual. I can say that I know I broke a personal commitment and that I feel like shit about it and have already said my 'sorrys' in person._ "

Quoth Danny Robinson, in a comment on Jamie Martin's blog: " _After everything that we did for you and Steven, I'm shocked at how you have twisted the truth for PR reasons. [...] in the end, I made sure that you got the better end (sic) the deal._ "

I can't think of two more opposite approaches to communicating about this matter... and I definitely know which one I prefer.

<comment><USER3> Humility is the best PR strategy in this kind of situation.

</comment></comment><comment><USER4> Here is another post by Boris: <http://bmann.ca/what-i-did-tonight>

Disclaimer: I've known Boris Mann for a while. He is a stand up guy. It's a crappy situation on all sides.

Missed capital calls happen during downturns. It's shitty for the investor that can't make the call. It's shitty for the VC that is running the fund. It's shitty for startup founders.

On a personal note, my company had a bunch of clients miss invoices in Q4 2009. Cash flow was horrible for that quarter.

Q4 2009 was crappy globally. It was extra crappy in Vancouver. A whole bunch of investors had cut deep in 2009 and the deepest cuts had been scheduled for Q4.

Anyone that had money had allocated it for February 2010. The Olympics were in town and everyone was doing their big push for that month.

I don't personally know Jamie of Status.ly but I do know others that were also let go and I do wish them the best.

<comment><USER5> Possible repsonses to "Q4 2009 was crappy globally":

Standard: It was the economy's fault.

Better: Times are tough so we'll have to work that much harder.

Best: A down economy is our greatest opportunity to put some distance between us and our struggling competitors.

</comment></comment><comment><USER6> _It all started with Jamie of Status.ly's post. I'm disappointed in lots of the original wording_

Has Jamie's original post been reworded? I've only come to it this morning (GMT) but it reads perfectly fine to me. Disappointed sure, a little worried about the future. I dont read it as blaming Bootuplab or having any anger directed at them...

(indeed it reads pretty factually to me; so I am assuming it has been reworded)

</comment><comment><USER7> I kinda wish Boris did the initial reply, as opposed to Danny Robinson and his "why art thou complaining?" comment. This sounds a lot more rational (though probably not as apologetic as I might've liked) than before.

</comment><comment><USER8> Are there any other incubators in Vancouver, ideally associated fairly directly with real (i.e. Silicon Valley, or at least Seattle) VCs?

If someone wanted to do just the minimum legal/financial/admin overhead for startups (say, $500/mo tops), and either provide them with YC-levels of funding, or provide housing/office space for 4mo ($1-3k/mo/person, tops), it seems like a win. Investing $20k in each company for 5-15 companies would be a good investment in dealflow for a Vancouver or global focused VC.

I think a Canadian funding entity investing directly could get around the crazy reporting requirements.

Even if you were only 20% as good as YC, open immigration laws of Canada, and the Vancouver environment would probably be enough.

Alternatively, a Vancouver franchise of Seedcamp or Founder Institute.

<comment><USER2> _Are there any other incubators in Vancouver, ideally associated fairly directly with real (i.e. Silicon Valley, or at least Seattle) VCs?_

Sadly no. If YC or TechStars set up in Vancouver I'd apply immediately; but Bootup Labs is the only show in town.

</comment></comment><comment><USER9> When reputations are at stake and someone got disappointed, it gets hard managing a PR crisis like this. But all Bootuplabs needs to do is stay focused, reach out to good developers and do its best to create valuable companies. People will pay for Jamie's experience if Bootuplabs creates winning startups that sell.

</comment><comment><USER10> One lesson to be learned here is that founders need to do their own due diligence on the firms from which they accept investment.

</comment><comment><USER11> I must say, just like his scummy partner Danny and his "we're disappointed in your, well, factual account of us dicking you" nonsense, I do really like Boris' little legal threats: "I read another round of comments that said that we should be commenting in public at all. I kind of know how to use these tools, and engaging where the discussion is taking place seems to be the right thing to do. _Our instinctive response is not to call in the lawyers_ [emphasis mine]… because, you know, we're actually entrepreneurs / startup guys ourselves." BootupLabs definitely brings the class.

If he and his partner had just been up front and said, you know, one of our investors failed to show up with cash as promised and the whole situation is a nightmare that we feel awful about, it would have put a very different spin on events. Along with, say, giving the startups they left hanging in the wind some support to get home and restart / transition their lives to the next thing. Instead, giving them a couple days to find a job in Canada? Fucking _awesome_. Entrepreneurs would do well to note how these people treat folks; I've never heard stories of YC acting like this.

<comment><USER2> _I do really like Boris' little legal threats_

I'm sure you're misinterpreting what Boris wrote. I read that line as saying that their instinctive response is to engage in a dialogue with their critics (even if IMHO their initial response was very poor) rather than running to a lawyer who would almost certainly advise them to say nothing.</comment></comment>