Hacker News's Undocumented Features and Behaviors - <AUTHOR> https://github.com/minimaxir/hacker-news-undocumented <comment><USER3> More and more websites are getting rid of user comments on their websites. Just in the last couple of months the National Review and The Atlantic have got rid of comments.

In simple terms they couldn't keep up with comment moderation and were not able or willing to invest in enough moderators.

So I have to give credit to HN to having one of the most civil comment sections on the internet. What I like about HN is that the comments VERY RARELY descend into the inevitable political sniping that seems to happen almost everywhere else on the internet, even when discussing controversial topics like Trump, and even the percentage of snarky and dismissive comments is kept pretty low.

So, keep it up, dang and sctb!

<comment><USER4> The amount of civility on HN is almost unbelievable. The behavior of the users is one of the HN's biggest strengths.

<comment><USER5> I once said something slightly political and my comment was downvoted into oblivion.

So that helps.

<comment><USER6> My political comments are some of my most popular ones I've made. I think it's more about saying something that's actually constructive or different. I think also the fact that hackernews is not ideologically segmented has to do with it, too. We're here for programming and decent, thought-provoking reading, not having people tell us we're right. You can get away with being political if you're genuinely trying to contribute to the conversation.

<comment><USER7> > actually constructive or different. > not ideologically segmented

i dont think that's really the case. HN is full of middle/upper middle class programmers who works in well-developed countries. This is definitely a segment on the political spectrum. This is why you don't see many trump voters here, or bigots or the very conservative. Anyone who attempts to bring discourse with such an undertone will get downvoted.

I'd say HN has more groupthink than the people here would care to admit.

<comment><USER8> > HN is full of middle/upper middle class programmers who works in well- > developed countries. This is definitely a segment on the political spectrum.

No, it's not. While class and profession have some loose correlation to ideology, the class/profession combination you describe includes people from all over the political spectrum, including all of the ones you say are excluded from HN for this reason (whuch, contrary to your description, seem to be well represented; well, not overt bigots but HN moderation is aimed pretty directly against that, and perhaps merely Trump _apologists_ , who may or may not be Trump _voters_.)

<comment><USER9> Sounds like an empirical question to me. I'd ante a moderate sum though that @chil's assessment is closer to the truth in terms of homogeneity of political leanings on HN and, if you could somehow measure it, that there is "...more groupthink than the people here would care to admit" on HN.

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> A few more of interest:

• You cannot downvote a comment after 24 hours have passed. (There is no time limit for upvotes.)

• You cannot downvote comments which are direct replies to your own. (There is no restriction to downvote further descendants.)

• You cannot unvote/undown a comment after (I think) one hour has passed.

• You cannot reply to a reply to one of your comments for a certain number of minutes, which increases based on some relation with nesting depth. However, the reply link appears sooner on the permalink of such comments than it does from the story page.

• You cannot edit a comment after (I think) one hour has passed.

<comment><USER11> _You cannot downvote a comment after 24 hours have passed._

This is a nice feature. One reason I stopped using certain subreddits was that people would look up your comment history and downvote all your comments that weren't archived. I think those downvotes would eventually disappear, but it annoyed me enough that I stopped reading the subreddits where it happened.

<comment><USER12> Then people on HN are just being lazy, by not writing software to auto- downvote all _future_ comments by people they dislike.

<comment><USER13> nice idea for a saas.

<comment><USER12> AFAIK, HN has no OAuth2 or similar, so a SAAS would not work for this task; the users would have to host such software themselves on someplace that they trust with their HN password.

<comment><USER14> If you tell me your password and I downvote everyone you hate, why is that less of a service than it would be if, instead of telling me your password, you gave me a special password with fewer permissions?

<comment><USER12> The first service comes at the cost of losing one's entire account. The second one does not. Therefore the second one is far more valuable.

</comment></comment></comment></comment></comment></comment></comment><comment><USER15> > There is no upper limit on the score of a comment, but the minimum score is > -4 points.

According to the _The Unofficial Hacker News FAQ_ [1] this is only partly correct, but I don't feel like testing it.

[1]

> I'm at -4 for a comment and still my karma is dropping, how come?

> HN is 'subtle' in that not everything always is what it seems. Votes are not > always counted and those votes that are counted are not always displayed. So > even though your vote count for that comment shows as -4, in actual fact it > may be much lower. Take your lumps, analyze why you got those downvotes and > if you feel they were justified then try to do better in the future. A side > effect of the votes still being counted past -4 seems to be that people tend > to delete their comments with a much higher frequency than in the past to > cap the damage to their karma.

[https://jacquesmattheij.com/the-unofficial-hn- faq#badcomment](https://jacquesmattheij.com/the-unofficial-hn-faq#badcomment)

<comment><USER14> I can confirm that the minimum display score is -4 points regardless of the amount of karma lost on a comment.

<comment><USER16> I think what your parent is getting at is that you'll lose at most 4 karma regardless of how many downvotes a comment may eventually accumulate. At least in my experience I haven't seen my karma fall more than 4 points due to a single comment, as far as I can remember. Have you a different experience?

<comment><USER14> Yes, of course. Downvotes are not shown (past the 5th), but karma losses are. Just as described in my parent comment:

>> I'm at -4 for a comment and still my karma is dropping, how come?

<comment><USER16> Wow. How recently? IIRC, that faq (though helpful) is out of date in sections. I assumed that that was one of them.

<comment><USER14> Not all that recently, but I see no reason to assume the behavior has changed.

<comment><USER16> And you're probably as willing as I to test its current status :). Cheers for clarifying this for me.

</comment></comment></comment></comment></comment></comment></comment><comment><USER17> A nitpick about notation:

For me, "Shadowbanning" or "hellbanning" means that the post of the user/site are [dead] but the systems lies to the user and to nim/her show the post as alive. And the only way to notice is to logout or get another computer and check the post from there.

Killing the post form an user or site automatically, and showing to the user that the posts are [dead] is not shadowbanning. I call it "autokilling" but I'm not sure that it is he official name.

Almost all the recent case/discussions of similar cases were simply autokilling. In the initial times of HN it was more common to have hellbanned users, but it's very rare now. Moreover, I remember a comment of dang where he said that they were considering to remove all the hellbanning and only use autokilling. I'm not sure if they finally made that change.

<comment><USER18> > Moreover, I remember a comment of dang where he said that they were > considering to remove all the hellbanning and only use autokilling. I'm not > sure if they finally made that change.

Nope, that change has not been made. I'm hellbanned.

<comment><USER19> Not really. Hell banning is a silent action. You don't tell the user they were banned.

You got several warnings, and then you were told you were banned.

[https://news.ycombinator.com/item?id=14938232](https://news.ycombinator.com/item?id=14938232)

[https://news.ycombinator.com/item?id=14960370](https://news.ycombinator.com/item?id=14960370)

EDIT: just for clarity: I'm not a mod.

<comment><USER17> @Kenji : You have a sibling comment that now is [dead]. Do you see it marked as [dead] or it looks to you like a normal comment? How do you know that you are banned?

[EDIT: Someone vouched your comment, so I see it now as a normal comment. I was going to vouch it after the experiment :( .]

Just to be 100% sure that we are using the same definitions: Can you post a screenshot in imgur of your sibling comment, so we can see how it looks to you?

For comparison: I see [https://imgur.com/a/1Z3dF](https://imgur.com/a/1Z3dF)

PS: Please don't downvote the GP comment, so we can have a nice conversation. Try to keep it black, or at least dark grey.

<comment><USER18> I'm too lazy to make a picture, but I see them solid black like a normal comment, there are no markings like [dead] or anything. Even for that sibling comment. For me it all looks normal except that my comments are always at the bottom.

</comment></comment></comment></comment></comment><comment><USER20> > Both are very responsive when contacted at email@example.com

True in my experience when I asked for my comment to be deleted.

> Complaining about being downvoted is discouraged and usually results in even > more downvotes.

It works sometimes however. My comment [1] was controversial and was moving up and down between negative and +4 votes for the first two hours of its life. After I made the edit complaining about the downvotes, it started rising rapidly and is now resting at +22.

> Hacker News encourages a single discussion on a given story. All others are > marked as a [dupe] and will be killed without the ability to vouch.

An addendum is that from my experience, it is unclear which story the mods will consider the dupe, sometimes the earlier posted story will be called the dupe.

> Relatedly, moderators can also invite users via email to resubmit a post > which didn't get much traction.

This happened to my "All the goodness gone from tea (1688)" [2] submission. I am amazed that a moderator read a submission that was only +2 in its original form and also had difficult ergonomics because it was on google books.

> One popular "trick" for obfuscating voting manipulation ... This trick > doesn't actually work.

So, undetectable vote manipulation hasn't been detected and there fore doesn't exist? I have trouble believing this. This seems more like a call to not even try claiming that it won't work, because it just might work. I wish some charitable soul with gray morals would step up with their story of voter manipulation that wasn't detected. Tooling could be developed to curb its use by other individuals. They can do this privately to the mods.

[1] [https://news.ycombinator.com/item?id=15994458](https://news.ycombinator.com/item?id=15994458)

[2] [https://news.ycombinator.com/item?id=15324547](https://news.ycombinator.com/item?id=15324547)

<comment> T-hawk > It works sometimes however.

Tone counts for everything when complaining about downvotes. Asking politely where you think you went wrong often does work. If it's hostile like "wake up sheeple", you're digging your hole deeper.

<comment><USER21> Indeed. Asking politely for what went wrong or for counter arguments usually helps. But it does not always.

Two points I got downvoted for without much substantial dicussion was when I said that security through obscurity does work (to some degree) and that tinkering with crypto is okay for everybody, and with certain preparation, can even be used in production in some cases. I stand by these points and are open to debate, but for the big majority I just get silently downvoted or arrogantly spoken to as if I didn't knew what I'm talking about.

I think this is a pity because tinkering with stuff is an essential part of being a hacker.

</comment></comment></comment><comment><USER22> My favorite little-known HN feature (not even mentioned on this link!) is the noprocrast mode. You can set in your profile to be locked out of HN for a set amount of time, after another set amount of time on HN. Prevents procrastination :)

<comment><AUTHOR> noprocrast is in the FAQ, which is why I didn't add it.

<comment><USER23> Maybe you should add "There's an FAQ" as an often-unrecognized feature.

<comment><AUTHOR> It's linked in the intro paragraph.

</comment></comment></comment></comment><comment><USER24> Also, IDK if this is mentioned anywhere, but HN comments have a maximum character limit. Once I made a veeery long comment [1], had to split them up in three pieces (which when I downloaded them using the API, take up 7.5kb, 6.5kb and 5.9kb respectively. I guess 7.5kb is a good approximation, but I did not try to make an accurate measurement.

[1] [https://news.ycombinator.com/item?id=15373653](https://news.ycombinator.com/item?id=15373653) This was about the decline of schools in my country, so I guess I'm going to be excused for this much of logorrhea :)

<comment><USER25> Wow that is _long_

<comment><USER24> :D I can talk about this topic for days, only stopping to hydrate and eating every twelve hours. It's my pet peeve.

</comment></comment></comment><comment><USER26> "after users reach 500 Karma, they gain the ability to downvote another comment"

Ahh that's why I don't see a downvote button :) - I thought it was a feature reserved to super privileged users

<comment><USER26> But not it made me wonder, how a positive enforcing behavior is only allowing down vote until you reach a threshold. It makes you engage and understand the community better before making your judgment.

<comment><USER27> In my opinion, it creates an echo chamber where in order to be a "full member" of the community you have to already espouse views the community supports. It is a big reason why I don't participate much.

<comment><USER28> I espouse a lot of views that the community does not support. I am also the first woman to make the leaderboard (under a different handle -- my retired handle has 25k karma).

I don't think it works exactly the way you think it does.

<comment><USER29> How do you know you were the first woman to do so? This is a psuedo anonymous internet forum where people aren't required to state their gender or even their real names.

<comment><USER28> [http://micheleincalifornia.blogspot.com/2017/06/how-do-i- kno...](http://micheleincalifornia.blogspot.com/2017/06/how-do-i-know-im- number-one.html)

I will add that being the first woman on the leaderboard is not essential to the point that you can get a lot of karma and not be part of the echo chamber. But inevitably that is the detail that gets focused on, as if proving that I am not the first woman (or questioning my assertion, because no proof to the contrary has ever been offered when I get asked this) somehow invalidates my actual point.

My other handle has 25k karma. This one has over 3k. I don't in any way whatsoever participate in an echo chamber.

<comment><USER29> > I will add that being the first woman on the leaderboard is not essential to > the point that you can get a lot of karma and not be part of the echo > chamber. But inevitably that is the detail that gets focused on,

If it's not essential, and you don't want people to focus on it, why did you bring it up? It was the only interesting part of your comment (to me). I don't really care if HN is an echo chamber or not.

> as if proving that I am not the first woman (or questioning my assertion, > because no proof to the contrary has ever been offered when I get asked > this) somehow invalidates my actual point.

I actually dont care if you were or were not the first woman or if it was someone else, I was just curious about how you knew on an anonymous forum. But go ahead, be mad about it.

Your blog post is interesting, so thanks for linking that.

</comment></comment></comment></comment></comment></comment></comment><comment><USER30> This is a great resources, but i'm forced to laugh at this bit:

> However, as of recently, the line between technology and politics has become > extremely blurred.

> as of recently

We really, really need to require CS majors to take ethics classes. Everything you do as an engineer has an impact on the world around you. Everything. This is actually really good, because most of us are good people! But you can only escape "politics" by escaping people and you simply will never do that.

<comment><USER31> Right on both counts, that it can't be separated and that it isn't recent. The OP needs to amend "The Guidelines state that political discussion is off- topic". The actual guidelines say _most_ and _probably_ , precisely to indicate _not all_.

Political articles that are on topic are those that contain material of intellectual interest, aren't just garden-variety politics or ideology, and contain substantive new information. Those are comparatively rare, but better able to support substantive discussion. Sometimes they get more upvotes than flags, or we may turn off the flags. There are plenty of examples of this—so many that we hear the complaint "HN has been taken over by politics" at least as often as its inverse.

[https://news.ycombinator.com/item?id=4922426](https://news.ycombinator.com/item?id=4922426)

<comment><USER30> I think the greatest possible contribution to all of science that hacker news (the piece of software + the community) could provide would be to extend its general self-reinforcing conversational tone to include issues of conscience and the ethical, moral, and political implications of the subjects discussed here. We simply can't treat the two groups as separate.

<comment><USER31> That sounds great, but isn't doable. It only seems doable, because of the dynamic described here: [https://news.ycombinator.com/item?id=16438476](https://news.ycombinator.com/item?id=16438476). What would actually happen is that it would make the community nastier and dumber until it died. Whom the gods would destroy, &c.

The fundamental tradeoff: the better HN gets, the more people want to use it for things that worsen it. This puts a cap on how good HN can ever get. All we can do is find ways to wriggle out of that tradeoff here and there.

<comment><USER30> Well that's depressing XD

</comment></comment></comment></comment></comment><comment><USER32> Missing from that list is everyone's favourite feature:

      Using code blocks as block quotes so as to make them almost unreadable on mobile. Especially when the post is already indented a few levels.

<comment><USER33>
        Using code blocks as block c

What's wrong with that?

<comment><USER34> The default styling results in it having too narrow a view so it requires a lot of side scrolling on mobile devices. It gets really annoying for block quoted text past 80-90 characters.

<comment><USER33> I'm on android Firefox at the moment and I can confirm that it becomes problematic after about thirty characters. That's what I meant by my previous post.

<comment><USER35> Unofficial HN community guide: Short comments meant to be viewed as sarcasm end up being interpreted incorrectly often when used in a serious forum. A short clarifying rider sentence explaining your real opinion often helps. That may ruin the joke, but if your comment was just to make a joke, it probably wasn't strong enough reason to comment anyway (for additional info on jokes, see the section entitled How to make a joke and not get downvoted).

Addendum: Poe's law is alive and well.

(The above is of course my take, and other people will follow their own best practices)

<comment><USER36> Sarcasm marks (e.g. "/s") are another option, although I was able to get it without the annotation.

</comment></comment></comment></comment></comment></comment><comment><USER37> I was _hoping_ to find all the unlisted URLs some of which I can never remember, like /leaders etc.

Interesting nonetheless.

There does seem to be some attempts to game HN lately that I don't remember in the past - like a recent tendency of some new accounts to ask dozens upon dozens of questions. I can only assume there's some karma gaming behind that.

<comment><USER38> > I was hoping to find all the unlisted URLs some of which I can never > remember, like /leaders etc.

[https://news.ycombinator.com/lists](https://news.ycombinator.com/lists)

<comment><USER39> Interestingly enough this doesn't include Hacker News Classic, one of my favourite frontends: [https://news.ycombinator.com/classic](https://news.ycombinator.com/classic)

<comment><USER40> I don't understand the difference between this and the regular front page. I can see it has different content, but why? Does it use a larger time factor? It seems like everything there is older, but maybe that's just because it calculates less frequently?

<comment><USER39> From what I can remember, it uses an older ranking system which is less correlated with upvotes/popularity.

<comment><USER31> It uses the same ranking system, but only counts votes from very early users. To me the interesting thing is how little it differs from the regular front page, although I think the explanation is disappointingly boring: the main predictor of upvotes—for all users, early or not—is already being on the front page.

<comment><USER40> It sounds like you guys basically left it in as a cross-check to make sure the community isn't deviating too far from the original premise. Very cool.

Thanks for the explanation!

</comment></comment></comment></comment></comment></comment></comment><comment><USER41> Question on "hidden" features: Sometimes sub-level comments (not even particularly deep) don't have a 'reply' button - is this turned off for particularly controversial (in voting) comments?

> Hacker News encourages a single discussion on a given story. All others are > marked as a [dupe] and will be killed[...]

The addition to this is that after a while this isn't enforced - which is why you'll see the same articles coming up reasonably regularly.

<comment><USER23> I've noticed the missing reply button; my impression is that it won't appear right away after a post is submitted, which is probably intended to reduce flamewars.

<comment><USER42> It's still possible to reply directly, but you have to click the permalink above the specific comment (the one with the time in it - "X minutes ago") before the reply button/box are visible.

</comment></comment></comment><comment><USER43> Not sure if this fits with the purpose of your page, but there are "easter eggs", like in the "Ask HN: What has HN given you?" thread( which is an awesome read [https://news.ycombinator.com/item?id=16409768](https://news.ycombinator.com/item?id=16409768) )

I noticed a MailGun cofounder mentioning the "HN Cofounder wanted spreadsheet" at [https://docs.google.com/spreadsheets/d/1Sygd1fhGYRS- ZvRP0IVV...](https://docs.google.com/spreadsheets/d/1Sygd1fhGYRS- ZvRP0IVV6rf3OUyED9_b6Da4tVWdS08/edit?toomany=true#gid=9) which is how he ended up involved with MailGun

EDIT:

this probablly isn't suitable for this page. I've started a page to track these things for myself at [https://github.com/keithn/HackerNewsCommunity/](https://github.com/keithn/HackerNewsCommunity/)

If anyone has any other gems that are worth tracking I'd love to know about them.

<comment><USER44> sorry if I'm being ignorant but how is that an easter egg?

<comment><USER43> It's not in the sense it's not built into HNs website, but it's an easter egg of the community in that you wouldn't know about it and nothing really documents things like this specifically related to the HN community.

It's like your page you are making, it's another easter egg of the community. Just disappears from view except for those who track it.

</comment></comment></comment><comment><USER45> I've noticed when using a VPN, my posts would be shown in my own account but not anyone else's. Thought this was a strange behavior.

Wasn't certain whether this is a consistency issue or whether posting through a VPN isn't possible.

Anyone know why? May be good to add it to the document.

<comment><USER46> To be clear, are you saying if you're on VPN you're shadow-banned, effectively, but the same account works fine when not on VPN?

<comment><USER16> I seem to recall 'dang mentioning this can happen, and that they can whitelist accounts. If you see it happening to yourself, you can contact them and they'll look into it.

</comment></comment></comment><comment><USER47> It's also a good idea to review the guidelines every once in a while:

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

</comment><comment><USER48> Something I noticed this week for the first time is the date on profiles changed from x days ago to the specific date a user's account was created. It's also now a link to the frontpage on that day.

<comment><USER31> We just did that. It seems like a neat way to tell people about [https://news.ycombinator.com/front](https://news.ycombinator.com/front), plus it's amusing to look at how HN was on someone's 'birthday'.

We also made older timestamps say "May 4, 2009" instead of "3124 days ago".

<comment><USER48> I do miss the x days ago, cracking the 3k club was nice.

<comment><USER31> I'm not entirely sure about that change either.

</comment></comment></comment></comment><comment><USER49> One thing I think it should have mentioned is that although comment scores are not shown to anyone but the user, it is easy enough to infer them, at least approximately. Obviously, negative scores are shown by desaturation. And positive scores are shown by sort order.

<comment><USER46> I hate the desaturation, I should add a user.css entry for it or something. Effectively silencing non-mainstream views seems so contrary to hacker culture to me.

<comment><USER50> Contrarian views aren't usually down voted here, low quality posts are. Troll posts that may have some semblance of an actual point should be rephrased to actually be part of a discussion.

<comment><USER46> I disagree - the are at least 2 top level comments on this story now that are greyed out. I consider them to be well-made, definitely not flames or spam, clearly written but contrary to the apparent majority view.

Even uncontroversial statements of fact get down-voted.

Take this:

>"Also, I don't think vouching for [dead] comments actually does anything beyond warm fuzzies." //

assuming the commenter didn't lie then it is a fact, they started their thought. Now the down-voters may know vouching revives comments, but that's no reason to downvote, that's reason to respond with a source.

Even if they are mistaken, it's harsh to downvote a view that can't be easily corrected because of hidden information.

Meh.

<comment><USER51> Yea, I just finished a comment about this very topic before I read yours. People here will downvote anything. There's a big voting audience, and chances are there will be at least some people who disagree with you on any topic. Best to just roll with it.

</comment></comment></comment></comment></comment><comment><USER52> What about [https://news.ycombinator.com/best](https://news.ycombinator.com/best) ?

<comment><USER53> And [https://news.ycombinator.com/active](https://news.ycombinator.com/active)

<comment><USER54> It's scary to think I've been here long enough to know things others don't.. Or maybe "those that would know won't tell." Either way: [https://news.ycombinator.com/leaders](https://news.ycombinator.com/leaders)

<comment><USER10> All the above are accessible from the "Lists" link at the bottom of most pages.

</comment></comment></comment></comment><comment><USER55> I'd like to see the second chance pool made public, if only as another entry under /lists.

<comment><USER56> That's already there, it's the new pages beyond the first one.

</comment></comment><comment><USER57> One question that's not answered: what do green usernames represent? Would honestly love to know!

<comment><USER58> As far as I know, new users are green for a certain time period.

edit: under 2 weeks old

<comment><USER59> I thought earning enough karma would adjust the color to default, I guesd I am wrong.

</comment></comment></comment><comment><USER60> I didn't realize you could see all topbar colors that have been set on the site. Would be a really interesting way to implement message passing.

</comment><comment><USER61> @minimaxir for your list: \- Not possible to downvote some posters, as the authors have extreme karma (e.g. >10,000, though I don't know the exact number), even if you have >>500 karma.

It is notable that HN does not support blocking particular users, or indeed annotating that you like them. Though plenty of fans will upvote well known authors, it is not possible for you to keep a list of people who you think have written well in the past. I'd love it if I could, e.g., mark favourite author names in green.

HN is also notable from my perspective for having some people with good technical sense and clear writing, but very extreme views on other matters, to the extent that they would be pariahs in RL situations.

<comment><USER62> I've wanted a feature to "follow" certain users as well. Presumably it could be built as a separate app or into a browser extension.

</comment></comment><comment><USER63> How does one contribute to the code running HN? I read the site almost exclusively on my tablet, and I would love a "open links in new window" settings option and would be quite happy to help write the code to make it happen.

<comment><USER64> This shouldn't be the default ever. I the user, have the control of I want a new window/tab or use the current one. Sites opening links in new windows/tabs try tomorrow me on their site, for their reasons, not mine. When building a web application: Make sure to not play with links in a way that Ctrl+click, context menu or, on mobile, long touch don't work, but allow me to open new tabs, and keep the state if I go away and then use the back button to come back. Even for non-powerusers this is a good habit. Non-powerusers are often surprised bu new windows popping up.

<comment><USER63> Never said it should be the default, just an option (like it is in Reddit). Also, one would hope even the below average HN reader wouldn't be surprised by a new window.

</comment></comment></comment><comment><USER65> Has anybody analysed how karma works for submissions that make the front page? It feels like there's either a rate limit on how much karma you can earn from a submission, or the amount of karma earned per upvote is inversely proportional to the number of upvotes, or something.

I've noticed that I earned 1 karma for 1 upvote for the first few upvotes, and also 1 karma for 1 upvote when the article's been off the front page for days but people are still reading and occasionally upvoting it. But in between it's definitely not a 1:1 - maybe more like 10:1 or even less.

It feels like there are probably some really interesting rules involved :)

<comment><AUTHOR> To my knowledge, upvotes which are suspected to be inauthentic do not count toward the overall karma gained (I have no citations, which is why I did not include it in the list)

<comment><USER35> Do you ever notice that you'll lose two points on a comment at the same time, then later they'll come back? I see that relatively often (once every couple weeks), and on completely uncontroversial comments, which is how I started noticing it. I wondered if it was some vote manipulation detection, where they've linked two accounts voting similarly and nullified their upvotes, but later clear them so they votes are reapplied. In any case, it's always two together (which is why I don't think it's an accidental downvote), and seems to be close to the same time if not the exact same time.

</comment></comment></comment><comment><USER66> "Discussions about race/gender" is also political discussion, it's very strange that the author doesn't understand this unless he has some implicit intentions here and separates these topics on purpose.

<comment><AUTHOR> A good example of the Hacker News reaction to gender topics (and no relation to partisan politics whatsoever) is what happened when a YC engineer launched a online community for women: [https://news.ycombinator.com/item?id=16159808](https://news.ycombinator.com/item?id=16159808)

</comment></comment><comment><USER67> > One popular "trick" for obfuscating voting manipulation on Hacker News is to > link to the Hacker News's /newest page of new submissions (instead of a > direct link which would otherwise make voting manipulation obvious), and > asking friends to upvote the submission from that page. This trick doesn't > actually work.

How does this not work? I am failing to see how legit people visiting new can be distinguished between people who are told to go to new and search for {{$title}} and upvote.

<comment><USER68> I noticed that too. The voting ring detector may only use the landing page component as a part of the overall algorithm? Other factors likely include the time since posting and common shared upvotes?

</comment></comment><comment><USER69> One thing which is not clear to me: I just recently had a submission which had more than twice of upvotes than had been added to my Karma. As far as I understand voting rings may be responsible for that. I am however much too lazy for engaging that way (I also have no social media accounts), I spent my time rather on HN ;-). Does someone has an explanation for this? Do others engage in those rings to get favorable submissions onto the fron page?

<comment><USER31> It gets complicated, but [https://news.ycombinator.com/item?id=16440811](https://news.ycombinator.com/item?id=16440811) has the basics.

<comment><USER69> Better view the parent of that post: [https://news.ycombinator.com/item?id=16438849](https://news.ycombinator.com/item?id=16438849)

</comment></comment></comment><comment><USER70> Great list minimaxir, thanks for that effort.

I'd love to know what the user activity is (over time zones) over a 24 hour period?

Obviously, YC is biased towards USA West Coast time (due to their location), so would be interesting to know how much user submissions vary by UTC time?

Also, I assumed that usernames in green type meant that those users had engaged/replied to one of your posts/submissions. But 'green' users - as in new - makes a lot of sense (once you know the 'rule').

<comment><USER62> I've wondered about the distribution across timezones for posts, comments, and votes as well. The HN data set on BigQuery might be helpful here.

[https://cloud.google.com/bigquery/public-data/hacker- news](https://cloud.google.com/bigquery/public-data/hacker-news)

[https://news.ycombinator.com/item?id=10440502](https://news.ycombinator.com/item?id=10440502)

</comment></comment><comment><USER71> >Implicit Downranking of Politics

The free pass given to certain genres of political content show which demographic lines HN is mostly homogeneous on.

</comment><comment><USER72> I feel like this is not enforced enough

Implicit Downranking of Politics

The Guidelines state that political discussion is off-topic. However, as of recently, the line between technology and politics has become extremely blurred. Most tech related submissions with a hint of political partisanship will quickly be flagged to death by users (or die a slow death due to the inevitable flame war).

<comment><USER31> The guidelines don't state that, as you can see by reading the second paragraph of [https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html) and, to quote pg from long ago, noting those words 'most' and 'probably'. ([https://news.ycombinator.com/item?id=4922426](https://news.ycombinator.com/item?id=4922426))

</comment></comment><comment> another-dave Think it would be interesting to include something on what the community expect from "Ask HN" and "Show HN" posts — thought I'd double-check the guidelines/FAQ before posting one before & realised they're not mentioned there at all.

Also, would be good to call out what flavour of formatting is supported by comments.

</comment><comment><USER73> Well now I want more karma so I can customize the top bar.

</comment><comment><USER74> Please don't over feature-ize it. This is about as good as it gets, for a simple text-predominant mechanism. I could wish for some things, but I could also expect pushback for their consequences.

So ... well done I guess. You gots it about right.

</comment><comment><USER75> > _and moderators can set a comment to automatically be collapsed if necessary > (e.g. meta-discussion)._

I'd like to be able to turn this off in my profile. Or maybe it can be linked to "showdead".

<comment><USER76> You could send an email to email@example.com requesting this feature. They're very responsive.

</comment></comment><comment> d--b Something I frequently use is search HN on algolia.

<comment><USER77> That's builtin (bottom bar of hn)

</comment></comment><comment><USER78> They also have a "fun" half way house before shadowbanning, limiting your comments to X pr Y hours...

</comment><comment><USER79> My latest campaign has been to find a way to add the tiniest bit of accountability for downvotes. I believe at a minimum the total number of downvotes given should be shown publicly on user profiles.

As things function now, downvotes are for all intents and purposes anonymous.

<comment><USER80> Why do any votes need 'accountability'? Their purpose is to identify comments worth reading. Almost everything people write about voting itself is not worth reading and the less of it, the better.

<comment><USER79> My intention is to ensure contributing to publicly censoring/censuring costs something (intentionally tiny, but greater than zero/nothing) each time, beyond the one-time minimum karma requirement. "Downvote to disagree" seems unlikely to scale indefinitely, particularly as mobile means less willingness to contribute beyond clicking up/down (specifically now that there are apps catching on that remove all "fat-fingered web UI" friction!).

It's perfectly fine to disagree with this belief; HN itself does!

[Edited]

<comment><USER80> I know it's perfectly fine, I'm trying to follow this (very common) line of thought and adherents never seem to explain it in a way I can understand.

For one thing you're not really 'censoring' anything but even if we say, for the sake of argument, that you are why should it have some cost different than the cost of 'promoting' something when you upvote? The mods' argument that meta is bad for actual conversation is also quite compelling. What's the counter to that other than stuff that mostly seems to boil down to some version of 'getting downvotes kind of feels bad' (of which calling downvotes 'censoring' seems like a particularly overwrought variant).

<comment><USER79> Thanks for taking the time to bring up these counterpoints!

> _why should it have some cost different than the cost of 'promoting' > something when you upvote_

If this is a blocker, show both! My recommendation to add the tiniest bit of friction only to the anonymous negative was an attempt to reflect existing site guidelines. My apologies if terminology is a distraction; I did add "censuring" as a better word.

> _mods ' argument that meta is bad for actual conversation_

Up/down votes are currently only public on the receiver's side, and this meta gamification powers HN. Would revealing the other side of the equation deter abuse of anonymous control (with an acceptable level of side effects)? All I can answer for sure is that intervention for downvote abuse (if any) is behind-the-scenes for now.

[I assume] the pool of eligible downvoters continues to grow; maybe when the time comes the next step will be another moderator rather than crowdsourcing -- the precedent of hiding individual comment scores seems relevant.

</comment></comment></comment></comment></comment><comment><USER81> To anyone interested in reading Hacker News source code, it is available here: [https://ycombinator.com/arc/arc3.tar](https://ycombinator.com/arc/arc3.tar)

<comment><USER76> It should be noted that that's an old, old snapshot of the code. The currently-live HN codebase is closed source.

</comment></comment><comment><USER82> There is also the "Slow down, you're posting too fast" feature that kicks in, likely to calm down flame wars, Unfortunately it seems to block everything, not just the thread in question.

<comment><USER31> That usually means your account is rate-limited. Various explanations at [https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...](https://hn.algolia.com/?sort=byDate&dateRange=all&type=comment&storyText=false&prefix=false&page=0&query=by:dang%20rate%20limit)

We're always happy to take the rate limit off if people give us reason to believe that they want to use HN as intended and will abide by [https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html) in the future.

</comment></comment><comment><USER83> I'll add that I've also seen the [flagged] tag appear on submissions that aren't dead. I assume it means that it's been flagged heavily enough to downrank it but not to kill it.

</comment><comment><USER84> >All comments start with a score of 1 point (but in order to prevent bandwagoning, the comment score is not visible to users other than the author).

How great would this be for news stories on Facebook?

</comment><comment><USER85> To me, the quality of discussions across various forums is:

HackerNews > Quora > Medium(comments) > Reddit > Facebook > Twitter > everything else

And these days, I use only the first 3 :)

</comment><comment><USER86> Great article, I learned a lot. I've wondered for a while why some usernames appear green, it would be nice to include that too.

<comment><USER87> New accounts are green. Seems that two weeks is the threshold for showing up normal.

<comment><USER35> I think moderators can bypass that as well. I'm fairly sure I've seen accounts go from green to normal within hours when it was the author of some paper or project that someone submitted finding out they are on HN and making an account to answer questions. I suspect moderators authenticated the user separately, or used the comments to do so if it was completely obvious.

</comment></comment></comment><comment><USER88> Slight nitpick: I believe downvoting is available >= 501 karma, i.e., above 500.

</comment><comment><USER89> It would be nice if 5x duplicates didn't show up in the RSS feed.

</comment><comment><USER90> Thank you Max! Well done.

</comment><comment><USER42> One more - flagging and vouching have an invisible weight to them, and I'm about 90% sure that weight is based on how well your flags and vouches line up with what everyone else does.

I've been here for years, and it used to be that when I flagged things, they went dead instantly. After a particularly controversial thread or two, not so much.

Vouch is the same way - one time I vouched a comment that was pretty plainly insulting, but had something that I thought was valuable for discussion in it. I vouched it, it went alive instantly, I replied, and it was apparently later flagged again. My vouches no longer appear to do anything.

I hate all this cloak and dagger stuff, to be honest. I've more than once stopped myself clicking "flag" because I know that I'll be penalized if I get it wrong (read: enough people disagree with me). And all that stuff above? It's pure speculation, because it's all the UI gives me.

Worse, barring the occasional bit of moderator invention to unkill a post, all of these mechanics serve to enforce a hivemind, whether or not that's the intent. Quality doesn't matter as much as consensus, because the consensus is enforced transparently, the quality with intervention.

<comment><USER91> I've had the same experience.

I thought about not flagging anymore, but so what. I'm offering my ratings, and if HN doesn't want to use them, so be it. And it feels good to flag shitposts.

</comment></comment><comment><USER92> This is pretty neat, thanks!

</comment><comment><USER49> Also, I don't think vouching for [dead] comments actually does anything beyond warm fuzzies.

<comment><USER93> I think it does.

I have, in the past, seen [dead] comments that (IMO) shouldn't have been, clicked on "vouch", and when the page refreshes they are no longer [dead].

Or maybe HN just shows it that way to me and it is, in fact, still [dead] (kinda like hellbans).

<comment><USER94> I've noticed that sometimes [dead] comments don't seem to be revived by - presumably - just me clicking vouch. Perhaps vouches and flags and other criteria are balanced against each other or something like that.

<comment><USER95> The OP article explains that vouch only works once, and if there are more flags it stays dead.

</comment></comment></comment></comment><comment><USER96> This is a FANTASTIC resource! Bookmarked! Thank you!

</comment><comment><USER97> I would love to see something like this:

      (Steve Jobs Died) (14 articles, newest on 11th of fyftember)

Articles on the same _event_ within short timespan would be compressed into a single line you can expand by clicking.

One implementation could be some kind of special purpose tagging functionality, similar to flagging, but with a keyword argument. Maybe the tag should even have a limited lifespan - until you can no longer assign it. Submissions under the same tag would be compressed into an expandable list, similar to threaded comments. Please make the chat flood stop!

<comment><USER98> This doesn't directly solve your problem, but if you click through to the comments section of an article and click the "web" link (between "past" and "favorite"), it'll search Google with the text of the post submission.

So if you're looking for multiple viewpoints, this is a ~frictionless way to start reading multiple perspectives.

<comment><USER97> My problem - hinted by the term "chat flood" \- is that some of the events don't interest me at all. I want to hide redundant submissions!

It's times like those when I'm reminded how US-centric HN is. I understand Steve Jobs is an American hero, but many people outside US only want to know that he died.

Some major security vulnerabilities fall into this category too. Usually some people with deeper understanding need to take care of it and push the changes upstream. I only need to know to watch out for X or Y hardware or service Z.

"Vote on net neutrality" type political submissions - you need to be an US citizen to vote.

</comment></comment></comment><comment><USER99> Just a side note, but... why are people using github as a content publishing platform? It just isn't a good UX for that, and there is no lack of workable CMS systems out there.

<comment><USER100> It could be multi-fold:

* It's free

* People are already signed up for it

* It brings exposure to your github profile

<comment><USER94> * github profiles are rarely deleted and content put up once will stay there for many years

Seriously, dead links to github.{com, io} are super-rare IME.

<comment><USER101> Aren't they deleted if they're inactive for N years?

<comment><USER76> I don't believe this is the case. They may become eligible for reclamation if someone else wants the name, per the Name Squatting Policy [0].

[0] [https://help.github.com/articles/name-squatting- policy/](https://help.github.com/articles/name-squatting-policy/)

</comment></comment></comment></comment></comment><comment><USER102> I've always been amazed that a proprietary engine with unclear rules and unavailable source code is the dominant source of news for the hacker community. Although it is designed better than reddit, unpopular opinions are still punished, which can stifle discussion and results in users self- moderating themselves and preventing themselves, even subconsciously, from expressing certain opinions from fear of downvoting.

Many of those undocumented behaviours are described with speculation - <USER2> don't we have a clear image of the capabilities and inner workings of Hacker News available anywhere? Security by obscurity?

<comment><USER46> Back when we had scores shown most people would seem to vote towards their ideal score for that post -- that was the consensus in voting discussions I was party to (IIRC). I'd vote down highly voted comments that I agreed with because the comment wasn't _that_ good; also upvote any comment that was negative that seemed to have value.

But then I learnt pg said basically 'downvote if you disagree', which IMO makes the site much worse. Also then votes were hidden and so I spend most votes on things I disagree with because they made a good point and I believe others should at least be able to see the text of the comment.

I also dislike hidden censorship, hidden rules, and hidden moderators/moderation (eg covert post promotion for Ycombinator companies).

<comment><USER19> pg said different things about downvoting. The last comment I link to seems to agree that some downvoting is a problem.

[https://news.ycombinator.com/item?id=658683](https://news.ycombinator.com/item?id=658683)

> IIRC we first had this conversation about a month after launch. Downvotes > have always been used to express disagreement. Or more precisely, a negative > score has: users seem not to downvote something they disagree with if it > already has a sufficiently negative score.

[https://news.ycombinator.com/item?id=117171](https://news.ycombinator.com/item?id=117171)

> I think it's ok to use the up and down arrows to express agreement. > Obviously the uparrows aren't only for applauding politeness, so it seems > reasonable that the downarrows aren't only for booing rudeness.

> It only becomes abuse when people resort to karma bombing: downvoting a lot > of comments by one user without reading them in order to subtract maximum > karma. Fortunately we now have several levels of software to protect against > that.

[https://news.ycombinator.com/item?id=2403716](https://news.ycombinator.com/item?id=2403716)

> [...] Another problem is that people use point scores as a guide to voting. > It's clear from voting patterns that many if not most users vote not to > express approval or disapproval, but to cause the comment to have what they > believe is an appropriate number of points. If I didn't display points, > people couldn't do that. Perhaps that's not a problem. But if it turned out > that that's what voting was for, then this could break voting, which would > in turn break the sorting of comments, which would be a problem now that > there are so many.

[https://news.ycombinator.com/item?id=1057338](https://news.ycombinator.com/item?id=1057338)

</comment></comment></comment><comment><USER95> This is a great write up, and answered a question I had about comments which are automatically collapsed. I understand the frustration around the lack of politics/diversity discussions, but I agree with the way it's handled here. Those discussions need to be had, but do they need to be had _here_? They are divisive, and often leave lasting wounds which lead to the creation of factions.

Mostly though, the internet is short on places like HN, and long on places to endlessly discuss politics, usually in a very predictable circle. I don't believe that much good comes from anonymous text exchanges online where politics are concerned.

<comment><USER33> People like discussing things on hn because the quality of discourse tends to be (relatively) very high. Naturally people would especially want to discuss politics or anything else that's almost universally a shitshow in a place where they thing commenters are thoughtful and fair and level-headed. Of course it could well be the case that we have such an environment here _because_ debating politics is discouraged.

<comment><USER103> There's another phenomenon at play with political stories, I think. Much of what occupies the front page is sort of implicitly siloed. Rust programmers are the active writers in Rust threads. Data science programmers are active in ML or word vector threads. People interested in math are active in math threads. There's a story on the front page about Google's "TPUv2's". I don't even know what a Google TPU is, so you're unlikely to have to deal with my comments on that thread.

Not so political threads! Everybody is on equal footing in them. There's no "people with expertise relating their experience" and "people asking questions" and the rest of us reading with interest. Rather, everyone's broadcasting.

That's not inherently bad, I don't think; there are lots of other kinds of threads that are like that and they seem to be fine (for the most part) for HN's culture.

But politics is a particularly toxic combination of broad-spectrum and polarizing. There are never voices of reason or experience, the most active participants are often on a hair trigger, and participation is amplified because everyone's got an opinion.

<comment><USER104> > I don't even know what a Google TPU is, so you're unlikely to have to deal > with my comments on that thread.

Damn there goes our shot to hear what serious crypto people are _really_ saying about Google TPUs.

:P

</comment></comment></comment></comment><comment><USER105> It will be interesting to see how we look back on the moderators' decision to stifle political discussion on HN. In my opinion it is a horrible abdication of responsibility among technologists to simply hide any content that pertains to politics.

But it is fitting that the political monoculture of the valley would give rise to an HN moderation regime that includes aggressive censorship.

I love HN and have learned a great deal from the articles and comments, but censoring political discussion is a form of sides-taking and HN mods are unabashedly responsible for it. It's a truly shameful dark cloud in what is otherwise a vibrant and flourishing ecosystem of ideas.

<comment><USER31> > _to simply hide any content that pertains to politics_

That's so inaccurate a description that I wonder how anyone could arrive at it.

<comment><USER105> If political discussions that evoke partisan loyalties end up being classified as "flame wars" due to the definition in the article, then if those involved in the so-called flame wars are censured (throttled etc) then effectively the topic has been discouraged simply because the votes averaged lower than the number of comments. Over time users learn to ignore such topics.

The big goof up is in the idea that all discussions with widespread disagreement are harmful. It's an absurd view of manners akin to saying that discussion about the ethics of slavery is simply rude to participate in regardless of one's views and regardless of whether any of the comments are actually rude or disrespectful (or are even, themselves, partisan. The votes may be where partisanship gets introduced).

Slavery is a bad example because it seems morally obvious and thus the enlightened side seems obvious. But many issues today lack moral clarity which is why intelligent discussion of them is deeply important.

Politics can be messy but in many ways political topics represent the conflict between values as some values gain or lose prominence.

<comment><USER31> That's an argument about how HN should be. I'm stating a fact about how it is. More at [https://news.ycombinator.com/item?id=16442668](https://news.ycombinator.com/item?id=16442668) and [https://news.ycombinator.com/item?id=16443186](https://news.ycombinator.com/item?id=16443186).</comment></comment></comment></comment>


The probability of data loss in large clusters - <AUTHOR> http://martin.kleppmann.com/2017/01/26/data-loss-in-large-clusters.html <comment><USER2> Unfortunately, this is not a very realistic analysis.

Martin mentions HDFS in the first paragraph as an example. But HDFS does not use partitions or consistent hashing for data assignment. All replicas in HDFS are placed manually according to the placement policy currently in effect. Each block can have a completely different set of nodes. He mentions Kafka as "typically deployed in... JBOD configuration." But Kafka is actually typically deployed over a RAID configuration. JBOD support in Kafka is fairly new and still not often used. It would have been better to mention only the systems that this analysis is focused on (Cassandra might be one such).

This analysis leaves out a critical factor, which is that the distributed system will start re-replicating each partition once nodes are declared to be lost. Clearly, if this can be completed before any other nodes fail, no data will be lost. This is the reason why you want more partitions-- not less as this essay recommends. With more, smaller partitions, you can re-replicate quicker.

To see this, consider a thought experiment where there is one partition per node in a 5,000 node cluster. In this case, each gigantic partition will be replicated to three different nodes, and the nodes will be in lockstep with each other. But if one of those nodes is lost, only 2 other machines in the cluster can help with re-replication. It will be slow. In contrast, if you have 10,000 partitions per node, and you lose a node, you will be able to use the resources of the entire cluster for re-replication (assuming you have well-distributed partitions). This means 5,000 disks spinning, 5,000 network interfaces passing data, instead of 2.

<comment><USER3> _This analysis leaves out a critical factor, which is that the distributed system will start re-replicating each partition once nodes are declared to be lost. Clearly, if this can be completed before any other nodes fail, no data will be lost. This is the reason why you want more partitions-- not less as this essay recommends. With more, smaller partitions, you can re-replicate quicker._

In practice though, it's not uncommon for multiple machines to fail at once: maybe you lose power to a rack, or a network switch dies. In this case your partitioning scheme really does matter if you want to minimize the odds of data loss. Using a smaller number of larger partitions can decrease the odds of data loss (though, at the expense of losing a larger amount of data if/when you _do_ lose all N replicas of a partition).

Using many smaller partitions can indeed speed recovery, and sometimes this is the right tradeoff to make. But, suppose _any_ amount of data loss means you have to take your cluster offline and rebuild your entire dataset from scratch: in this case, you may be better off with a small number of partitions per node.

<comment><USER4> That's effectively a partition, not data loss. It is good to be rack-aware though for data placement to still be available in the case of rack loss.

</comment></comment></comment><comment><USER3> For a more in-depth analysis of these kinds of effects, see:

[http://web.stanford.edu/~skatti/pubs/usenix13-copysets.pdf](http://web.stanford.edu/~skatti/pubs/usenix13-copysets.pdf)

The authors basically try to determine "if all data is replicated to N nodes, how can we minimize the probability of data loss when a random N nodes go down at once?" There are some really interesting tradeoffs that arise in the process of answering that question.

</comment><comment><USER5> I remember reading, some time ago [1] [2] about how many errors were expected without a complete failure of the HDD. I mean, the odds that you just read a rotten byte out from your disk. And how that would be enough to cause data loss, since you don't need 3 complete failures to loose data: perhaps 1 complete failure plus 2 partial HDD failures...

And as HDD sizes grew larger, things would become complicate. To the point that the probability of loosing the second disk while still restoring the first one, since HDD sizes were so big that the time it takes to rebuild a whole 3TB disk is huge, in a environment that is in production, reading and writing things at the same time you're reconstructing the lost one.

[1] This is one of such articles: [http://searchdatabackup.techtarget.com/video/Preston-As- hard...](http://searchdatabackup.techtarget.com/video/Preston-As-hard-drive- capacity-grows-so-does-risk-of-data-loss)

[2] this is another: [http://www.zdnet.com/article/why-raid-6-stops-working- in-201...](http://www.zdnet.com/article/why-raid-6-stops-working-in-2019/)

<comment><USER6> Even with datasheet values -- rebuilding a RAID5 with 4 TB or even larger disks has a _significant_ chance of failure (iirc about 15 % for HGST MegaScale 4 TB drives; since the BER is pretty much a constant, it will be much worse with larger drives. RAID5 probably does not make much sense with 8, 10 or 12 TB drives.).

<comment><USER7> This is why RAID6 and similar are used. Or tree like structure of RAID10. Mostly because the probability of a HDD dying whole is lower than in part, you get much more of the extra redundancy than expected.

These should be also patrolled and resilvered regularly, which should catch imminent failures too.

</comment></comment></comment><comment><USER8> I'm surprised Erasure Coding didn't come up here. It greatly increases durability and storage space required at the cost of higher latency reads.

[https://en.wikipedia.org/wiki/Erasure_code](https://en.wikipedia.org/wiki/Erasure_code)

<comment><USER9> Erasure encoding does not greatly increase storage space consumption. Consider a 2n or 3n replication factor in systems like hdfs, Cassandra and Riak. In those cases storage space consumption is 200% and 300%, respectively. Erasure encoding allows you to select the percentage increase in space consumed for some level of durability. You can, for instance, confirgure the system to encode a given blob of data at 1.5x (150%) space consumption and then stripe that data across 10 or so locations, only requiring 6 or 7 of them to reconstruct the original blob. Facebook put a post out about this some while back re. their long term picture storage.

<comment><USER8> You're correct, I mistyped & intended to say what you wrote.

</comment></comment></comment><comment><USER10> He basically says that the chance of loosing one unit of data linearly increases with the number of units of data the cluster contains. That does not surprise me.

</comment><comment><USER11> The largest error in this analysis is the arbitrary picking of a .1% failure rate for some arbitrary time period. In order for this analysis to make sense, it should be right at the probability of failure for the time period it takes to replicate back to 3 copies. What makes this hard to compute is that drives don't have a linear death rate in regards to age, it's more "bathtub" or `U` shaped. And this number should really be a function of the ages of all the disks.

Drives seem to have a sweet spot of reliability after a few months and up until a 2-4 years that is really low.

</comment><comment><USER12> Has Backblaze done any digging through their data for hard disk failure correlations among manufacturers? I know some manufacturers show higher rates than others but within same model have they seen spikes of failures at around the same age, # of reads/writes, etc?

Are there any other large public hard drive failure datasets that could contribute to refining the type of analysis in this article?

<comment><USER13> They do statistics a few times a year:

[https://www.backblaze.com/blog/hard-drive-failure- rates-q3-2...](https://www.backblaze.com/blog/hard-drive-failure- rates-q3-2016/)

</comment></comment><comment><USER14> Even ignoring counter-intuitive probabilities, the law of large numbers is often surprising:

[http://dl.acm.org/citation.cfm?doid=347059.347561](http://dl.acm.org/citation.cfm?doid=347059.347561)

If you deal with large data, you might have experienced such issues already - hope you have application level hashes on top!

</comment><comment><USER15> Over time, data loss is the same regardless of cluster size. It's just that as cluster size grows, you go from "lose 100% of your data in 0.1% of time periods" to "lose 0.1% of your data in 100% of time periods". This analysis looks at only a single time period so that effect is obscured.

</comment><comment><USER16> He's leaving out a critical factor which is the replication time (in relation to the time used for the failure chance). The probabilities will be lower if you take it into account unless you have very long replication times.</comment>


WireGuard and forward secrecy for clients - <AUTHOR> https://dustri.org/b/wireguard-and-forward-secrecy-for-clients.html <comment><USER2> I work at IVPN.

WireGuard out of the box indeed have few issues related to privacy which require additional effort to overcome. Forward secrecy is one of them.

We created a document where we've described all of the known issues related to privacy when using WireGuard up to date and how we solve them [0].

Talking about forward secrecy specifically, IVPN client apps for all platforms do automatic key rotation every 7 days by default (which can be changed to as often as every day).

It makes it impossible to correlate any traffic with customers after keys were rotated.

[0] - [https://www.ivpn.net/knowledgebase/254/Using-WireGuard- for-P...](https://www.ivpn.net/knowledgebase/254/Using-WireGuard-for-Privacy- Protection.html)

</comment><comment><USER3> Is this really an issue? The client's public key is likely unique for that server (unlike a typical ssh public key), and the client's IP is a lot more likely to be used to identify the user.

</comment><comment><USER4> This is a pretty big deal, PFS is a major feature in modern crypto (TLS, IPsec etc). Hope it gets fixed soon.

<comment><USER5> This is not about PFS for transmitted data, only for client identity. Unless I'm missing something, an attacker will not be able to decrypt past traffic if they manage to obtain the private key, but they will know the public key of whoever sent it.

In most cases this is a non-issue, but it can matter of you're trying to hide your identity behind a VPN.

</comment></comment><comment><USER6> Is a lack of PFS still a problem if you periodically retire and regenerate keys?</comment>


Ask HN: Is there an open source equivalent to Palantir? - <AUTHOR> It seems to me that journalists and activists could benefit from software that could help track networks, timelines, etc., and organize unstructured data the way an intelligence analyst would. Anything like that out there? <comment><USER2> I've been working on a project called Aleph ([https://github.com/pudo/aleph](https://github.com/pudo/aleph), live: [http://data.occrp.org/](http://data.occrp.org/)), which is targeted at investigative reporters. It's supposed to handle a related set of problems, which is data integration of diverse public records and journalistic lead generation (i.e. what to investigate next). The road map for the coming two years includes some visual analytics similar to what I guess most people use Palantir for (we'll have to keep it very simple).

The big problem in NGO space is obviously finding engineers: we've got some budget for this stuff, but getting people to join us, work at a lower salary and then code their heart out (because teams are tiny) is hard.

On the fun side, you get to see your code applied to fighting real-world problems every day :)

<comment><USER3> I would love to try out your tool, but I'm getting a 504 Gateway Time-out on [http://data.occrp.org/](http://data.occrp.org/). Is there any chance I can give Aleph a spin?

<comment><USER2> Yeah we have a scheduled maintenance right now (hello, Docker friends!). Should be back in 10-15min. There's also some other instances:

[http://aleph.openoil.net/](http://aleph.openoil.net/) [http://opengazettes.org.za/](http://opengazettes.org.za/)

</comment></comment></comment><comment><USER4> I worked at palantir. As a designer. Grains of salt.

The value is in the dataset. Most of what's on top is handmade by fresh grads.

Spend your effort getting access to the firehose, and the rest will follow.

<comment><USER5> Orly. The value of Palantir you think is in the dataset but NOT in the full stack 10 ish years old low technical risk analytical platform?

<comment><USER6> there isnt any 10 ish year old low technical risk analytical platform. The wheel is remade for everycustomer, palantir is not a software company but a services one.

</comment></comment></comment><comment><USER7> I think you're asking the wrong question. There is nothing special about Palantir software that you can't find in the open source world already. What Palantir does is just throw a bunch of bodies at the problem to "integrate" things.

If you truly want to provide an open source alternative then curating some jupyter notebooks with the right libraries and integration is all it takes. A properly curated set of R libraries with a nice interface would also do the trick.

<comment><AUTHOR> I think people greatly underestimate the importance of integration, but I take your point

</comment></comment><comment><USER8> The only thing close that I can think of is: [https://github.com/sirensolutions/kibi](https://github.com/sirensolutions/kibi)

Demo video: [https://www.youtube.com/watch?v=g0O8UNM0B7Y](https://www.youtube.com/watch?v=g0O8UNM0B7Y)

</comment><comment><USER9> The Investigative Consortium of Journalists used Neo4j to untangle the Panama Papers leak.

[https://neo4j.com/blog/icij-neo4j-unravel-panama- papers/](https://neo4j.com/blog/icij-neo4j-unravel-panama-papers/)

<comment><USER10> The problem with Neo4j is that the end results are great, but the ingestion pipeline (especially for unstructured data) is very hard to make general purpose.

The ICIJ used a combination of Apache Tika, Nuix, Tesseract and a bunch of other components when loading data into Neo4j before interrogating it within Linkurious.

It's also worth noting, that Panama data-set is riddled with data quality issues (even if this is understandable given the size of the team compared to the scale of the problem).

<comment><USER11> My first job out of university was with a Palantir competitor (Detica at the time I was hired, then BAE Systems Detica, then finally BAE Systems Analytics or something like that[1]). There's nothing general purpose about either of their platforms (nor the similar offering from SAS). Companies like that just throw a bunch of fresh graduates at the data, and they hand-write loads of custom ETL code for every data set. A lot of times, even the "analytics" are just shitty little pattern matches over tiny subgraphs formed from the data, the vast majority of which are also coded anew by the "analysts"[2] for each data set. Data quality issues were handled with a massive case analysis during ETL.

There's really no magic going on in such products — the only part that's really general purpose is the GUI used to view the end results. The rest of it is just a bunch of lowly peons doing a _ton_ of gruntwork to hammer the data into a form that said GUI will accept.

[1]: That last name change was after I'd left. I didn't even make it a full year at the company before my conscience got the better of me and I quit.

[2]: An "analytics" job at Detica was really just a half-step above data entry. It was mind-numbing and soul-eating. There was a very high turnover rate because even new graduates were overqualified for the position, and almost everyone was miserable.

</comment></comment></comment><comment> m-i-l In the UK we have [https://fullfact.org](https://fullfact.org) , an independent fact checking charity. It isn't quite the same thing, e.g. is more of a service than software, but is open source friendly, e.g. was the topic of a recent Lucene hackathon.

<comment><USER12> Brilliant landing page at present, I must say!

</comment></comment><comment><USER13> Most of the core platforms these companies use are built on open source. I think the fundamental reason there isn't something comparable in the free and open arena is because at the end of the day to coral the data volumes you will find your self well on your way down the road to hell.

There is a minimum physical hardware cost in the multi-million dollar range just to get started.

Not to mention the labor cost to operate and architect said hardware. Any thing in the Petabyte scale is looking at a minimum skeleton crew of (if they are top talent, more people if they aren't top talent) maybe 3-4 people between sysadmin architect and data engineers to keep things production worthy. The industry is still light on people with those skills so ontop of the bare hardware you are probably talking about 600 to 800 K in annual labor cost as a starting point.

This would all be before you get that much usage, the minute that utilization becomes high that skeleton crew won't cut it any more. The SysAdmin will need to turn into an on-call ops team. Data Engineering will need specialists for on-boarding vs analytics vs access layer.

If you are doing anything controversial don't forget the importance of a solid security organization which is highly challenging in the distributed computing space.

These are just the technical hurdles. Depending on the data you would be bringing in there are quite possibly legal barriers as well depending on the locality.

So again its not so much that this isn't possible, more that there hasn't been someone willing to endow the kind of funding that would be required to scale something of this nature solely for the benefit of the public good.

<comment><AUTHOR> Nah. I'm thinking of the people trying to understand the relationship between (for example) Trump and the Russians. They don't have that much data because they're not drinking the firehose of wiretaps. They're relying on news reports, contacts, financial documents, etc. It's a question of helping people connect the dots.

</comment></comment><comment><USER14> I would start looking at gephi: [https://gephi.org](https://gephi.org) and sigma.js for visualization. To run influence or social network analysis I would look at R's igraph: [http://igraph.org/r/](http://igraph.org/r/) or spark graphx: [http://spark.apache.org/docs/latest/graphx-programming- guide...](http://spark.apache.org/docs/latest/graphx-programming- guide.html#the-property-graph) . For the graph database. neo4j , elasticsearch or the open source graph database created by the uk intelligence service: [https://github.com/gchq/Gaffer](https://github.com/gchq/Gaffer) . Another interesting project is mazerunner that integrates spark graphx and neo4j together : [https://neo4j.com/developer/apache- spark/#mazerunner](https://neo4j.com/developer/apache-spark/#mazerunner)

<comment><USER15> You may want to look into Linkurious ([https://linkurio.us](https://linkurio.us)) which integrates Neo4j, ElasticSearch and a graph visualization interface. It was used extensively in the Panama Papers ([https://linkurio.us/panama-papers-how-linkurious-enables- ici...](https://linkurio.us/panama-papers-how-linkurious-enables-icij-to- investigate-the-massive-mossack-fonseca-leaks/))

Jean from Linkurious

<comment><USER14> That looks interesting. Out of curiosity Is this open source , how does that compare to maze runner ?

</comment></comment></comment><comment><USER16> We are starting a program at Faraday ([http://faraday.io](http://faraday.io)) to offer complimentary access to journalists and educators. Faraday certainly isn't Palantir, but it's similar.

If you're a journalist or educator and want to check it out, shoot me an email, I'm andy@

</comment><comment><USER17> I dunno, are there non-state actors with the capacity to inflitrate and disrupt militarized adersaries at a global scale in furtherance of the foreign policy motives of a super power on the world stage?

I wonder if they'd release under the GPL3 or or BSD licence. Maybe they'll let me fork them on github.

<comment><USER18> I'd like to find some software to help me manipulate the world order without being detected. Any cool projects I should look into?

</comment></comment><comment><USER19> Not exactly what you're asking for. But [http://www.jplusplus.org/en/](http://www.jplusplus.org/en/) have mailing lists and is a community of journalists looking through data. I'm sure they have alternatives or suggestions.

<comment><AUTHOR> I'm thinking about software that does link analysis - this person works with that person, got paid by other person, etc. Basically, computer support for intelligence analysts, but in the public service.

</comment></comment><comment><USER20> Are you certain that you want a competitor to palantir? It sounds like you might want I2 Analyst Notebook from IBM, or something like maltego from Paterva. Maltego is also free (ish) which is cool.

<comment><AUTHOR> This is what I'm looking for. Thank you.

</comment></comment><comment><USER21> Because that capability is too valuable. So if you're interested in writing code that can do that you can find someone to make your life comfortable on the condition that they get to keep your code. If repo of code starts growing that can do that someone will come offering them candies :-).

As it gets a bit more common place, or if tools to do some of the heavy lifing get out.

Keep an eye on some programs like the one at CU Boulder, it should produce some interesting research that moves this along outside of the halls of industy.

</comment><comment><USER22> I recently release a project of mine that may be of interest: "Gransk - Document processing for investigations" ([https://gransk.com](https://gransk.com) and [https://github.com/pcbje/gransk](https://github.com/pcbje/gransk))

</comment><comment><USER23> Given their pricing (and the pricing of nearest competitors), combined with the relatively solved technical problem areas they operate in (connect to a data source, do some human driven NLP, draw a graph, show a map, map everything to some ontology), what they do is very ripe for some serious open source disruption.

The easy part is really the interface and information displays, the harder part (and where they make the lions share of their money) is in data connection services and software customization.

Building an Open Source Palantir tool wouldn't be all that difficult, in fact a great many organizations just build some subset of that tools using readily available open source components and with tighter coupling to their business needs. But these efforts are fractured and disorganized and there isn't a great centralized open source tool that really replicates their system.

Should there be? I think the general problem of pulling together lots of information into a common pool, then being able to annotate that data and map it to a semantic model is useful, and it generalizes well. But at the same time, many many sources of data are already available in nice semantically organized ways, with simpler interfaces (think IMDB, Pouet, Wikipedia, etc.) it's not quite clear that their approach offers enough payoff over these easier methods.

</comment><comment><USER24> GDELT project provides open data on everything that happens on the planet.

</comment><comment><USER25> You can start here: [https://en.wikipedia.org/wiki/Social_network_analysis_softwa...](https://en.wikipedia.org/wiki/Social_network_analysis_software)

<comment><USER26> In addition to those on the wikipedia link from Jonnybgood, see also

lumify (lumify.io) developed by Altamira

Visual Understanding Environment ([http://vue.tufts.edu/](http://vue.tufts.edu/)) from Tufts

Lumify is for networked data analysis and data annotation

VUE is more for presentation but has very basic network, and semantic analysis tools and ontology mapping capabilities.

and last but not least - TimeFlow is old but useful for mapping temporal sequences quickly and easily. See [https://drawingbynumbers.org/tools/timeflow.html](https://drawingbynumbers.org/tools/timeflow.html)

</comment></comment><comment><USER27> RTÉ Investigations Unit used OrientDB to find corruption cases: [http://orientdb.com/rte-iu_case-study/](http://orientdb.com/rte-iu_case- study/)

</comment><comment><USER28> There are companies out there(Sumologic is one) which provide free access depending on unstructured data volume. But in case, you have huge volumes of data in TBs and you want to manage it using open source, you might end up spending money in scaling and supporting system that you should instead buy paid versions.

</comment><comment><USER29> "[..] from software that could help track networks, timelines, etc., and organize unstructured data the way an intelligence analyst would."

Do you know how it looks in Palantir's case? I've never actually seen it ..

<comment><AUTHOR> no. they keep it pretty hidden, but there are other tools designed for intelligence analysis that are easier to see.

<comment><USER30> Well, for certain values of "hidden" that include 40 minute long demos of product on Youtube, e.g. [https://www.youtube.com/watch?v=f86VKjFSMJE](https://www.youtube.com/watch?v=f86VKjFSMJE)

Look around in their 293 videos on that channel and I'm sure you'll see a wide variety of their products, I just pulled out the first one that popped up and looked like a demo. Remember, companies need to advertise to find new customers, so they can't be too secretive.

(My biggest tip to someone looking to interview at a company is to find their youtube channel and watch a few videos of them demo their product before you interview. You will be an expert on their product compared to 95% of the people they interview, which will make you stand out. If they publish white papers, read one or two that look interesting to you. If they publish white papers and none of them look interesting to you, that's a pretty important sign too. Sure, bone up on how to find a cycle in a directed graph, but spend an hour or two on the company themselves and it will be very rewarding.)

</comment></comment></comment><comment><USER31> You might try Lumify from Altamira -- [http://lumify.io/](http://lumify.io/).

<comment><AUTHOR> The newer version is called visallo. [https://github.com/v5analytics/visallo](https://github.com/v5analytics/visallo). same people different company/platform

</comment></comment><comment><USER32> Its called Wikipedia [ [https://youtu.be/ESVQknHESuA?t=2209](https://youtu.be/ESVQknHESuA?t=2209) ]

The paid versions are called Google, Palantir, the CIA etc.</comment>


Tesla Model 3 Production Lower, Price Higher (A Lot): Morgan Stanley Analyst - <AUTHOR> https://www.forbes.com/sites/brookecrothers/2017/03/28/tesla-model-3-production-lower-price-higher-a-lot-morgan-stanley-analyst/ <comment><USER2> This is a great example of terrible writing. The analyst is predicting that the average selling price for Model 3, including options, is going to be much higher than the base price.

That's great news for Tesla, but the writer makes it look like it's a disagreement (it isn't) and doesn't explain the context (great news for Tesla.)</comment>