Facebook Plans on Backdooring WhatsApp? - <AUTHOR> https://www.schneier.com/blog/archives/2019/08/facebook_plans_.html <comment><USER2> We haven't added a backdoor to WhatsApp. The Forbes contributor referred to a technical talk about client side AI in general to conclude that we might do client side scanning of content on WhatsApp for anti-abuse purposes.

To be crystal clear, we have not done this, have zero plans to do so, and if we ever did it would be quite obvious and detectable that we had done it. We understand the serious concerns this type of approach would raise which is why we are opposed to it.

<comment><USER3> Sadly it boils down to "trust us" (or really, trust wcathcart), which is a position users have been betrayed in countless times over the past decades (and Facebook has a horrible reputation for user privacy). Compare that with Signal—or any other application with an open-source client—where we can inspect the source code and compile our own client.

<comment><USER4> > Sadly it boils down to "trust us"

If this is done client side, it doesn't boils down to that. You can easily decompile and see for yourself what it does. You will gain quite a bit of notoriety if you are the first one to catch them too.

As he said:

> if we ever did it would be quite obvious and detectable that we had done it.

<comment><USER5> > You [can] decompile and see for yourself what it does.

Assuming your device allows you to get the binary. Apple is already in a position to disallow this if they choose to in the future.

<comment><USER4> > Assuming your device allows you to get the binary. Apple is already in a > position to disallow this if they choose to in the future.

Theses kinds of thing never stopped anyone. Being the first to share a hash of a system file in a console is always an achievement that many hackers tend race to do when a new one is released.

For sure the harder it is, the less person will do it, thus the more theses things will be able to go under the radar, but for now it's not so much an issue.

<comment><USER5> For now. Until you need a tunnelling electron microscope to maintain the fiction that you still own the hardware.

</comment></comment></comment></comment></comment></comment><comment><USER6> This article cites a Forbes contributor (Kalev Leetaru) whose articles are endlessly self-cited with no outside sourcing. Can anyone confirm that any of this is actually real? I got 3 layers deep and could find nothing of substance.

<comment><USER7> Does it really matter? We already know that these companies are willing to collaborate with intelligence services. We already know that Facebook is hoovering up your data. We know Jan Koum already left Facebook over privacy issues. Maybe the article is click bait trash, maybe it isn't, but we should assume that whatsapp isn't secure.

<comment><USER8> Does it matter if things are true or made up? Yes.

<comment><USER9> Unfortunately, it's likely we'll never get a rigorous proof of malfeasance. Thus do you go with WhatsApp as a homosexual in Russia or Iran, or as protestor in Hong Kong against the Chinese government, given that Facebook has given us plenty of reasons to distrust them?

<comment><USER8> I'm not saying to trust Facebook. I think it is wise not to trust them. Just because I don't trust Facebook doesn't mean that I should assume by default that every allegation against Facebook is true.

</comment></comment></comment></comment></comment><comment><AUTHOR> This shows the downside of network-effects so succinctly. It's almost hopeless to think people will realistically move to a technically superior alternative without this privacy hole.

<comment><USER10> The nice thing about direct-chat apps is that the network effect isn't nearly as strong as for traditional social networks. If you want to use X to talk with someone, you can just download it and then talk with that person. You don't have to get everyone to switch at once. You'll have more little app icons on your phone but that's about the extent of the cost.

<comment><USER11> Probably 95+% of all my sent or received Whatsapp messages are for groups. If I don't get _most_ of the people I communicate with to switch to another (the same!) messenger, it's pretty inconvenient.

It doesn't help that, last I checked, the good (open) alternatives were all much jankier or missing one or more major features we use. Makes them hard to sell. :-/

<comment><USER12> Curious, what features is Telegram missing for you?

<comment><USER13> Default end to end encryption.

<comment><USER14> They were talking about group chats.

</comment></comment></comment></comment></comment></comment><comment><USER15> "source is based on overdrawn conclusions from a speculative article. The linked Forbes (F1) article goes to another Forbes article (F2), which links to the Developer talk. F2 is a speculative article based on the Facebook talk..."

References:

[1] [https://www.schneier.com/blog/archives/2019/08/facebook_plan...](https://www.schneier.com/blog/archives/2019/08/facebook_plans_.html#c6796641)

[2] [https://www.reddit.com/r/privacy/comments/ckrmjg/facebook_pl...](https://www.reddit.com/r/privacy/comments/ckrmjg/facebook_plans_on_backdooring_whatsapp/evqfez3)

</comment><comment><USER16> Personally I've felt this has been in the works for some time.

If you look closely at how the founders exited facebook, and what was said at the time, it seemed (to me at least) that they left over exactly this.

I mean, that aside, can you really come up with many other operational arguments that would result in people walking a way from hundreds of millions of dollars? (or by some estimates leaving over a billion dollars on the table between the two of them)

<comment><USER17> Confirmation bias is bad when others have it, but not when HN readers have it.

You haven't bothered to read the source, haven't bothered to understand what the context is, and haven't even bothered to see that there is no mention of WhatsApp in the talk.

Ducking tired of click bait and people who ducking reward it.

:face-palm:

<comment><USER18> You can in fact say _fuck_ if you want to. There are lots of offensive things said in pleasant tones in polite society. Bad language barely moves the needle so give me an F...

<comment><USER19> Yeah, changing it to _ducking_ actually shifts the reader's focus from "Wow this person feels very strongly about this issue" to trying to decide what the tone is meant to convey.

It took my brain a few tries to parse the word duck in this context.

</comment></comment></comment></comment><comment><USER20> > why does AT&T care about content moderation -- but it is something to watch?

"Content moderation" is just another kind of surveillance. You have to have been watching -- surveilling -- in order to moderate.

AT&T already has capital and people invested in surveillance. [https://en.wikipedia.org/wiki/Room_641A](https://en.wikipedia.org/wiki/Room_641A)

<comment><USER21> Content moderation is also censorship. You typically use E2E to avoid censorship and surveillance. Facebook wants to mold thought and opinion, it is the ultimate control over society.

</comment></comment><comment><USER22> There is new pressure in the UK to backdoor encryption and their target is facebook / whatapp.

[https://www.theregister.co.uk/2019/07/31/home_sec_priti_pate...](https://www.theregister.co.uk/2019/07/31/home_sec_priti_patel_five_eyes_encryption_controversy/)

So I am not surprised.

</comment><comment><USER23> I never understood government overreach like this. Any developer can clone an open source crypto implementation and spin up a new chat app. It is ridiculously stupid to backdoor things.

Unless the government plans to go back to the 80s, in which cryptography is heavily restricted from export, this doesn't accomplish anything.

<comment><USER24> > Any developer can clone an open source crypto implementation and spin up a > new chat app.

And they can use it to converse only with themselves. Getting everyone else to stop using their favourite panopticon is the hard part.

<comment><USER25> I disagree, and history does not show this to be true. We have been more fickle when it comes to instant messengers than almost any other technology on the face of the planet (pun intended). From IRC, ICQ, AIM, Facebook messages on the website (not Messenger), WhatsApp, Telegram. I count 6 major shifts in the last 19 years. This would indicate that the over/under lifespan of a messenger is roughly 3.5 years.

All it takes is a usable interface, general availability, and a reason to change. If WhatsApp starts doing content moderation, of any form on the client, I am of the strong opinion that users will find this to be "creepy" and will immediately start shifting. As it is, all of my technical contacts have already left WhatsApp for Telegram. Since that group/audience has been the bellwether for all of the previous instant messaging client shifts over the last 2 decades, I imagine it's only time before "everyone else" also shifts off WhatsApp. I don't necessarily think it will be Telegram, but they definitely have the most momentum today when compared to, eg, Line, Zalo, Viber, Kik, et al.

</comment></comment></comment><comment><USER26> That's the point!

We have to TRUST WhatsApp, Telegram, Signal, et al because there are no good OPEN SOURCE alternatives. Otherwise we could just run our own.

We have to TRUST the current PKI and DNS because the alternatives are still immature.

But EVEN IF we develop robust and mature alternatives, and wide adoption, we have to TRUST the Device, OS and Browser makers.

The Trusted Computing Base is made by a handful or companies. Open source in hardware is still a nascent field.

It's interesting if one can ever be truly sure that some part or chip hasn't been interdicted. Apple tries to scan components as they arrive vs their hardware designs.

But hardware will always be the weak link. Keyloggers. Cameras watching your fingers from the ceiling.

The future of surveillance is in this kind of stuff. We will ALMOST have secure communications, but not really. The only thing you can be sure of is sending quantum entangled particles from airgapped rooms.

<comment><USER27> > there are no good OPEN SOURCE alternatives

Yes, there are. There has been for many years. Matrix or XMPP with OMEMO.

</comment></comment><comment><USER28> my (albeit faint) hope is that this leads some ambitious (EU?) organization to spin this as a US tech/corporate/power overreach angle, starts heavily promoting Signal as an alternative, various security conscious orgs/govts buy in, leading to big Signal popularity boost

... hopefully

<comment><USER29> Forget this one, EU politicians, especially our German Interior Minister Seehofer, hate proper e2e with a passion - they want to weaken the 5g standards so that they can continue IMSI catching and snooping.

Fuck them hard.

</comment></comment><comment><USER30> Speaking of whatsapp.

Does anyone else think it's weird that it requires access to your camera roll to view images, access to your camera to use the web version, and access to your contacts to start group chats? (on android anyway)

Aside from the webapp QR-code thing, none of that is technically necessary.

</comment><comment><USER31> _INSERT Blake screaming at Alan and MacLeod in Highlander II: The Quickening_

"Please don't bother, Alan. Every time you turn that thing on it just gets printed out in my office. And yes, yes, yes, yes, yes, yes, yes' yes, that includes your little computer"

</comment><comment><USER32> I thought they already did this in Egypt and perhaps one or two other countries long, long ago.

<comment><USER14> Yes I remember that! It was developed in a joint venture between Nokia and Siemens. People got tortured because of that infrastructure. So, Nokia/Siemens said in a press release, "gosh we didn't know Egypt and Iran would do that if we would help them violate human rights now we are sorry" and fuck them both.

<comment><USER32> Indeed. And if they're willing to backdoor a regime like that then why could we ever trust them not to backdoor everywhere else?

<comment><USER14> Well, they both made a press statement, saying that they wouldn't!

</comment></comment></comment></comment><comment><USER33> Yes, Facebook, do that. Will make it so much easier to convince more friends to switch to Signal :-)

</comment><comment><USER34> Facebook has never cared about your privacy. E2E encryption should be a requirement for any communications app. Not a half truth.

I hope people begin to take notice.

<comment><USER35> And open source allowing for independent verification. We have tons of good open source apps like Signal and Matrix, why would we ever need one controlled by facebook of all companies?

<comment><USER36> > why would we ever need one controlled by facebook of all companies?

The value of WhatsApp is not that its technically better than Signal or Matrix, the value is that there are 1B+ users.

</comment></comment></comment><comment><USER31> > _The encryption . . . debate is already over – Facebook ended it earlier > this year.

[...]

After all, if either user's device is compromised, unbreakable encryption is of little relevance.

[...]

The problem is that if Facebook's model succeeds, it will only be a matter of time before device manufacturers and mobile operating system developers embed similar tools directly into devices themselves, making them impossible to escape. . . . effectively ending the era of encrypted communications._

I guess FB is one of the more visibly monstrous instances of surveillance capitalism so it works well as an example, but I don't understand why the author is acting as if what it allegedly aims to do is unprecedented. As if similar phenomena hadn't already been happening surreptitiously throughout various other hardware/software for quite a long time.

</comment><comment><USER37> Use Signal. The. End.

<comment><USER38> I like [https://wire.com/](https://wire.com/)

<comment><USER39> Unpopular opinions ahead. I like Telegram and Wire, with Telegram far ahead in features. Once I get people to use Telegram and describe how it's different and better than WhatsApp (except default E2E), they realize that it is indeed nicer, and want to reduce using WhatsApp.

Wire is fine (not as good as or fast as Telegram), but the company seems to be focused on paying customers to increase revenues. So free customers hardly get any support or responses from support (the auto-replies from Wire support state this priority). I like that Wire is by default E2E _and_ syncs conversations across devices and platforms!

I've been waiting to recommend Signal to others for at least three or four years now. But it's 2019, and Signal is still bad in UX and messaging reliability (not to mention one still can't migrate conversations to another device on iOS).

<comment><USER40> Somehow Telegram shows normal chats as notifications but if its an encrypted chat, it shows a "You received a message". Seems they are hostile to e2e.

<comment><USER41> No, they just don't want to leak the message contents to your notification screen.

</comment></comment></comment></comment></comment><comment><USER42> JACK: "No. It's unconstitutional. You can't make your own currency. That is the federal government's job.

PRICE: Jack. Look at me. I am not the problem here. The problem is that hard cash is fading. Rapidly. That's just the way of the world right now. And Bitcoin is spreading — and if Bitcoin takes over we are all in a world of hell. It is unregulated. It has already reached its transaction volume maximum and it is partly controlled by Chinese miners.

With E-Coin we control the ledger and the mining servers — we are the authority. _I will make sure that you will have visibility into every single wallet that's opened: every loan, every transaction_. Which means we can start making new assets. Which means we can start rebuilding the banking sector without you having to inject even more politically unpalatable federal funds into it.

JACK: The President will laugh in my face.

PRICE: But he will know that this is the right thing to do. This is going to be controlled by a good old-fashioned American company. You want to regulate it? Be my guest. _Regulate the shit out of it. I'll give you backdoors, side- doors, trace — whatever you want_. Just don't. Shut it. Down.

<comment><USER43> This is the first thing I thought of when I learned about Libra. It's _terrifying_.

</comment></comment><comment><USER44> I just don't buy that Facebook will only limit itself to what it tells us publicly that it will do.

Facebook has recently talked about merging Messenger, WhatsApp, and Instagram chat into one while "also keeping end-to-end encryption."

Why would Facebook cuts its own access to all the data ming it was getting with the Messenger and Instagram chat? It wouldn't, unless it uses "faux" e2e encryption. Besides, Facebook currently allows advertisers to send ads Messenger users. How would those ads be managed by FB's servers, if the encryption is truly end-to-end between the sender and the receiver?

Also, this "new encryption system" is exactly what the founders of WhatsApp left:

[https://www.nytimes.com/2018/04/30/technology/whatsapp- faceb...](https://www.nytimes.com/2018/04/30/technology/whatsapp-facebook-jan- koum.html)

<comment><USER6> My understanding of FB's plan to E2E encrypt everything is that it would be done to dodge responsibility for content moderation. If it can't see what's being discussed, it can't be held responsible for reigning in hate speech, misinformation, etc.

<comment><USER21> Who defines what is hate speech and what is misinformation though? We're in post truth where I don't even think the "good actors" are acting truthfully.</comment></comment></comment>


Ask HN: Where did the address prefix 192 come from? - <AUTHOR> On a private Lan, the addresses start with 192 (ie 192.168.0.1).  The number is set by the rfc.  Why was the number 192 chosen? <comment><USER2> [https://en.wikipedia.org/wiki/Classful_network](https://en.wikipedia.org/wiki/Classful_network)</comment>


A Go implementation of Poly1305 that makes sense - <AUTHOR> https://blog.filippo.io/a-literate-go-implementation-of-poly1305/ <comment><USER2> One reason this cannot be translated into a portable C version while retaining its compactness (and readability) is the lack of support for carry/borrow chains.

In Go, bits.Add64 takes as input and returns the carry.

In C, there are some constructs that modern compilers will recognize as patterns people use to extract the carry, but it is a hit-and-miss.

Even compiler-specific intrinsics are a few and far between, inefficient or even broken.

<comment><USER3> If you look into the implementations of Add64/Sub64/Mul64 you'll see that it's not a lot of code and that there is nothing magical about them. They can just as easily be implemented in C in some 30 lines of code. Because they concern unsigned integers only there is no UB to consider on overflow/carry.

<comment><USER4> Actually, those functions are intrinsics in the Go compiler on many platforms, so the compiler generates a single ADD / ADC opcode. The source code that you see is a fallback for backends that have not implemented them as intrinsics yet.

<comment><USER5> gcc can optimize generic code for a carry check. To be fair both clang and msvc fails to optimize this specific case.

[https://godbolt.org/z/6WXfnL](https://godbolt.org/z/6WXfnL)

Edit: OK, I have much worse time generating "adc" instructions, and only gcc sees the opportunity to emit "setc" where appropriate. I see the issue now.

<comment><USER4> Correct, gcc can generate "setc" but not "adc", AFAIK.

</comment></comment></comment></comment></comment><comment><USER6> Great write-up, I haven't dug into understanding the whole code yet, but perhaps someone can answer this for me: can the code be made even simpler using Go's bignum: big.Int?

<comment><USER7> One concern may be the lack of a constant-time guarantee. If there are any optimizations that run differently depending on the numbers involved, then that may be an opening to a side-channel attack.

<comment><USER8> Upvoted, but such optimizations might exist for integer math, too. I'm not aware of any compiler that gives constant-time guarantees.

The best you can do is to write statements that you think map to certain instructions, and to check every release build to see what instructions the compiler chose.

You also may have to make sure that the buffers you use don't cross page boundaries.

<comment><USER9> Unit tests that throw random numbers into the code and check execution time is always a constant number of clock cycles should catch nearly all timing side channels.

<comment><USER10> That won't work. When we say "constant-time code" what we really mean is "code whose running time is independent of its input". It won't really be constant due to factors like the OS scheduler and the initial state of the CPU's instruction cache.

What you really want is [https://github.com/agl/ctgrind](https://github.com/agl/ctgrind) or something similar.

</comment></comment></comment></comment></comment><comment><USER11> I find a lot of go libraries are very well commented and very accessible, but the standard library and especially the crypto stuff is very sparsely commented.</comment>


Ask HN: Market for SAS/R developers? - <AUTHOR>

So I was informed by my current employer that one of the things on their todo list is to have me do SAS and/or R programming along with the PHP/Java that I'm doing now. While it's great to have more languages on the ole resume, I'm wondering what the market for these languages are like? Is there crazy demand for them out there, or is it as useful as saying your'e a HTML programmer? <comment><USER2> Knowing SAS/R in isolation is worth very little. Knowing statistics and data analysis is worth a lot and SAS/R are pretty good tools for doing statistics and data analysis. Unless you're also willing to learn the underlying statistics and data analysis there isn't much demand for SAS or R programmers. However there is massive demand for competent data analysts capable using SAS or R.

</comment><comment><USER3> Not sure what domain you're in, but here's the view from mine...

SAS is extremely common in pharma companies. Basically, any company that has anything to do with a clinical drug trial will be using SAS.

In the basic research / tool development parts of the biotech industry, R is common as well. Today, basically every biotech company has some kind of "data science" component (though they won't necessarily have adopted that buzzword).

Either way, in my view this is a far more specific and valuable skill than "HTML programmer".

<comment><AUTHOR> Ya I should have specified my domain when posting this. I'm currently in the healthcare field for one of the larger hospitals here in Toronto. One of the reasons for this posting was to gauge how useful it'd be for the private sector, as it's both my preference and it's only a year long contract.

</comment></comment><comment><USER4> Many companies that employ Data Scientists in NYC use R for their stats and ML work. While I personally prefer Python over R for ML related tasks, R does seem to be gaining traction in the private sector. I recommend ML for Hackers if you're interested in learning more about R - the author is very knowledgable and works as a Data Scientist for Facebook.

[http://shop.oreilly.com/product/0636920018483.do](http://shop.oreilly.com/product/0636920018483.do)

</comment><comment><USER5> SAS and R are heavily used for data and statistical analysis. As such, they're very employable "languages." SAS especially can pull down big bucks, but R is the better language/platform. I worked with SAS extensively (and learned R in college) and really disliked SAS. It is widely used but felt very archaic and poorly thought-out. The interface is ugly as sin and the SAS language is a mess. After using it for a year it blows my mind that companies pay so much to use it.

But employable? Definitely.

</comment><comment><USER2> I have a couple of friends who do portfolio and risk management at two different pension funds, and both of those places use a lot of SAS.

That being said neither of my friends knew SAS when they where hired and I doubt there are many places that hire people because they know SAS. They hire people because they know statistics and risk analysis and then send them on a 4 day SAS course if necessary.

</comment><comment><USER6> SAS is huge in financial services. I have come across it quite a bit. I have statistics knowledge, but not programming skills really. I know of many jobs requiring SAS. I have never seen a HTML programming job listing. R is mostly used in academia. I do not know of anyone in the private sector using it.

</comment><comment><USER7> If you want to go into more statistical type of roles in the future (data science, analytical marketing, etc.) than they're very useful.

Outside of that they're of less use.</comment>


A new type of book - <AUTHOR> http://aenism.com/end-of-passive/ <comment><USER2> I don't believe that reading is a "passive" activity.

"[Tom] saw a new girl in the garden--a lovely little blue-eyed creature with yellow hair plaited into two long-tails, white summer frock and embroidered pantalettes." (Tom Sayer, by Mark Twain)

Reading this, we all pictured the girl in our minds. But I believe the exact image of the girl I have is different from yours. This is our mind working and filling in the gaps of the information that it believes to be missing. Some people may see the girl vividly, filling it with details like some freckles and an embroidered dress while others will simply see a blonde girl with blue eyes. But our ming WORKED to make that.

In the book Understanding Comics by Scott McCloud, he says that the most import place in a comic book is the place between the pannel, where our mind rushes in to fill in the gaps, to connect the action.

Even movies have this. We have the jump cut, that can be considered a jump from one panel to another (with the implicit "space in between"). We also have the "places outside the scene". If someone goes away to get a coffee, we can picture it happening. If the scene is being filmed in a room, and we can only see 3 of the walls, we KNOW there is a fourth one.

And, besides it all, with every medium we consume, we have the "baggage" we take with us. Our knowledge of previous stories, movies, songs, books; our own opinions on the theme and even if we had a good or bad day will influence our experiencing of the "object" (book, movie, etc.)

It isn't because we're not moving our hands that the medium becomes "passive".

Books are what they are. And they are good at it. The author's fault is that he is trying to change books while what he should really be doing is creating a new medium.

</comment><comment><USER3> I think this is really interesting, though in many ways it sounds more like a game than a book. A book (to me) has a single story line based around a structure which the author intended you to follow. Even a 'choose your own adventure' had a basic story the user followed. I admit, technology now allows us to now have as many stories intertwine as we like, it still doesn't make it a reimagination of the book.

It does make the reimagination of a story. What I find most facinating about this idea is how it relates to an amazing skill I've noticed in two of my friends. Whoever they meet, within a few minutes, they are able to ascertain what makes that person tick, what they're interests are, and what makes them special.

If you're picking up a story and intertwining with different characters at different points, you'll need to be able to do that in order to make the story interesting.

<comment><AUTHOR> A reader will always follow a path and every path is a single story. Like how you are not aware of other parallel universes. So I still think it's more of the hardware that's being reimagined. You just flip the book differently.

About your two friends, it sounds like intuition. How do you think it relates?

</comment></comment><comment><USER4> BBC Radio 4Extra has run a couple of experiments that feel like early iterations of some of those ideas.

One was a drama where listers could vote on a couple of out comes at the end of each episode.

The other was a crowd-sourced drama. They'd start with an episode, then invite listener-written followups. List ers would write each episode. (The episodes were short and the plots got twisty fast).

But the idea in the article sounds like how some games should be. It'd be great if games could use more, better, writers.

I'm not sure how easy it would be to allow a person to follow a path and not have clashing incongruities when different paths merge.

</comment><comment><USER5> I've been thinking along these same lines. It's going to be very tricky to simultaneously get the rich language required for interesting reading and make it sufficiently interactive. It will be pushing the state of the art in natural language processing, because portions of the story will need to be procedurally generated as you go. Or possibly have a team of writers modifying the story in real time as you go, which is also intriguing. To make it interesting and interactive, there will most likely be too many branches to do it any other way.

By the way, nice diagrams.

</comment><comment><USER6> He's trying to re-invent the MUSH without knowing what a MUSH is or the early experiments with MUSHes linked to the web and such.

From my experience with MUSHes in the mid-early 90s the biggest problem is the general public is not terribly creative. A general cultural desire for drama and formulaic entertainment. Look at how big the Second Life world was, but how the population density was about 75% clustered around the spawn point talking, regardless of the world being nearly limitless in size.

<comment><AUTHOR> What's a MUSH?

<comment><USER6> I'm not going to try to beat wikipedia and google at their game, but I can try something like what if your MUD / text adventure was more mod-able than minecraft?

The experience of playing around on them in the 90s reminds me strongly of the linked article's ideas.

</comment></comment></comment><comment><USER7> I can't see how the fact that other players can modify my story with their decision could be somehow more fun that a pre-scripted story. I mean, yes, it sounds somehow fascinating, but it doesn't really impact the fun on the game. And what about the tradeoff with the quality of the story? You can "easily" make 1 epic\awesome story, you can't do that with hundreds of forks.

</comment><comment><USER8> Like a visual novel? ([http://en.wikipedia.org/wiki/Visual_novel](http://en.wikipedia.org/wiki/Visual_novel))

<comment><AUTHOR> Except each character "instance" is controlled by another reader at each juncture.

</comment></comment><comment><USER9> Very interesting!</comment>


Stop Eliminating Perfectly Good Candidates by Asking Them the Wrong Questions - <AUTHOR> https://hbr.org/2019/03/stop-eliminating-perfectly-good-candidates-by-asking-them-the-wrong-questions <comment><USER2> tldr: If the lock is broken, it doesn't really matter whether the key is the right one or not.</comment>


When I Was Your Age - <AUTHOR> https://www.americanprogress.org/issues/economy/report/2016/03/03/131627/when-i-was-your-age/ <comment><USER2> Paraphrasing the classics. It is the housing market stupid.

Really, Even living in a place with very low population density(Rural Mass/NH). Rents are high (and scarce), you need at least 250K to buy a small house. Houses are old, Lot's are big, big distances, very few new houses are built every year. I can't even image how blue collar families do it. If you think about it, it even makes local companies less competitive, since they need to pay more so people can barely survive.

This resonated a lot with me. [https://medium.com/the-ferenstein- wire/a-26-year-old-mit-gra...](https://medium.com/the-ferenstein- wire/a-26-year-old-mit-graduate-is-turning-heads-over-his-theory-that-income- inequality-is-actually-2a3b423e0c#.iljcrro5m)

<comment><USER3> Our keystone national housing policy focuses on appreciating prices.

A sensible national housing policy would focus on improving the characteristics of the housing stock (improving energy efficiency, removing lead, etc). Of course the building code moves forward over time and new housing generally beats older housing on metrics like that, and there are occasionally tax incentives for making improvements. but none of that comes close to the mortgage interest deduction in scope.

It would also focus on affordability. It's clear enough that with 40 or 50 years for the policy to work and ~1/2 of the nation not having much of any wealth that home equity is not the path to universal prosperity.

<comment><USER4> This is one of the reasons that Germany housing policy is so successful: [http://www.forbes.com/forbes/welcome/#1ef0063ebb2d](http://www.forbes.com/forbes/welcome/#1ef0063ebb2d)

In the US, housing is treated as an _asset_ , which is a huge mistake. In place like Germany, most people are renters because: rents stay fairly low, the government reassesses population periodically to allow more houses to get built, and renters have plenty of rights. Here, we allow houses to act as a hybrid place to live, ATM, and stock.

<comment><USER5> Living in your investment is a bit like having your 401K funds invested in your employer: If something goes wrong, you're doubly-screwed.

</comment></comment></comment></comment><comment><USER6> >Millennials have spent almost their entire working lives in a labor market that is loose—with too many job seekers and too few jobs—and where private- sector labor unions are almost entirely absent. Certainly, monetary policy that promotes employment while making it easier for workers to form unions would help Millennials make up lost ground.

The issue I have with this is it promotes a certain ideology—that job-seekers have the right to hold jobs even when there are too few. This article completely ignores the other paradigm, that if you don't have a job, it's up to you to make yourself valuable to employers.

I'm not trying to argue for one ideology or the other, I just think it's absurd to ignore the existence of one or the other.

<comment><USER7> _...that if you don 't have a job, it's up to you to make yourself valuable to employers._

Most people do not have the opportunity to make themselves valuable.

It's generally understood that at this point in history most people shouldn't really have to work at all. The whole point of technology is to make living easier for everyone. The root of this problem, as far as I can understand it, is misallocation of capital, not of "jobs". The efficiency of the free market is telling the labor market, "hey, you don't need to do this job anymore." So why are we still forcing these people to work? They should be allocated the capital gains reaped from market efficiencies that are a result of technological innovations.

<comment><USER8> The basic problem is that this idea that people shouldn't have to work goes against centuries of religious thinking which is strongly embedded in American culture. The idea of the Protestant/Puritan work ethic is that hard work is a sign of God's grace and centrally important to the character of a person and society as a whole. Sometimes you will hear people talk about this directly, but more often it's just implied because it has become awkward to assert religious ideas on society explicitly.

Until that cultural pillar of hard work defining a person's character and place in God's kingdom goes away, we will be stuck with this idea that people are defined by their work.

<comment><USER9> So think about this another way...

What gives you more self-worth, learning a new programming language and completing a project in it, or sitting on the couch and watching TV?

I personally believe the ancients understood this, from the usual parables. Idle hands are of the devil, and lead you to fill your time with something. Most of the easy somethings are drinking, eating and junk entertainment.

Technology may give us less to do, but at the expense of less opportunities for developing self worth. Even some of the founding computer members understood this--Wiener, a member of the "Dream Machine" crew mentioned it in the book "Cybernetics".

Perhaps I may clarify the historical background of the present situation if I say that the first industrial revolution, the revolution of the "dark satanic mills," was the devaluation of the human arm by the competition of machinery. . . . The modern industrial revolution [i.e., the computer revolution] is similarly bound to devalue the human brain. . . . The answer, of course, is to have a society based on human values other than buying and selling. To arrive at this society, we need a good deal of planning and a good deal of struggle. . . . (Wiener, 1948, pp. 37-38, bracketed words added)

<comment><USER10> Well, you're comparing "learning a new programming language and completing a project in it" with "sitting on the couch and watching TV". There are things you can do instead of "working" that isn't being a couch potato.

For example, learning how to play an instrument, picking up painting, attempting to master chess, or Go, are all things that you could do instead of working and are still personally enriching, if not financially enriching. They may not boost the economy directly but they will make the culture a better place nonetheless. There is a lot of untapped potential that could be explored if people have a guaranteed basic income and didn't _have_ to work that dead- end, minimum-wage job.

Of course we still need a lot of human labour to make society run. I mean who _wants_ to do menial labour, or plough snow at 4am so the morning commute isn't a complete disaster.

So I don't have an answer, I just think you're falsely equating "not working" with "idle hands".

</comment></comment></comment></comment></comment><comment><USER11> I'd be interested in seeing the change in wage distribution by profession. In the 70's it was still possible to exit high school and move into manufacturing jobs that paid 2-4x minimum wage. Those jobs are largely gone now, and the opportunities for 30-year-olds with high school only are more often (I speculate) Walmart or Starbucks or Chipotle type jobs. There is a higher percentage of college educated that may balance that out, but we need to see the distribution to understand it.

<comment><USER12> People I know that skipped college and began working and saving right after high school (all as waiters), were able to purchase a home, or a two-family, or three units for investment purposes with the rock bottom prices/foreclosures during the housing crisis.

They have been able to accumulate some wealth on service industry salaries.

They don't make much less than many of their counterparts with liberal arts degrees, who without the head start graduated into the recession with nothing.

For many the opportunity cost of college was high.

<comment><USER13> I dated a bartender for a few years and one of my best friends waits tables, so I've ended up meeting a fair number of industry people. I have no idea what you're talking about. They're almost all renters. And this is in St. Louis, where housing is cheap. (Also, I'm in my thirties. Not talking about people just starting out.)

<comment><USER12> What's cheap?

My cousin bought his first foreclosure for 33k during the south Florida housing bust. He has two more now, all from waiting tables and being extremely thirfty.

A 140k two family amortized over 30 years at 3.5% can be free to live in for the owner occupying one unit. That can be done in Connecticut for example.

This is how some of my non colege educated millenial friends are getting by quite well.

</comment></comment></comment></comment><comment><USER14> It's no surprise that the Center For American Progress pins the blame for this on lack of labor participation among Millennials, but I don't buy it. We're talking about a sub-2% drop compared to Gen X.

<comment><USER15> You forgot to adjust for inflation. 2% drop on money that is worth 6% less or so.

<comment><USER14> I was referring to unionized labor participation between Gen X (30 in 2004) and Millennial (30 in 2014) as defined by the article, meaning that its overall effect on Millennial income is negligible enough to approach zero.

Not sure that inflation in relevant to that.

<comment><USER16> Small absolute differences but that's still a ~22% decrease in union membership at the same age.. It could definitely affect wages, and certainly does for those in formerly union jobs.

</comment></comment></comment></comment><comment><USER17> Another issue worth noting is getting past the set requirements to be evaluated by a human. I'm 30, without a college education, and worked for Apple retail as a 'Genius' for 5 years. That meant enough at the time to get me a job at Hulu with the expectation of upward mobility. Unfortunately, Hulu's executives made poor decisions and the type of work I was doing was unbearable. After leaving, I've been unable to find a job due to my application submissions missing a key component to even be considered; a college degree. Now, no one cares I worked for Apple like they used to.

<comment><USER18> The whole college degree as a filter thing is the laziest thing a company could do to quickly get through a stack of applications. On the other hand the market is apparently in their favor enough that they can afford to use lazy filters like these that might lead to missing out on an awesome employee.

<comment><USER19> There really needs to be a way to get honorary degrees of some sort where you've put in the work. College degrees are something focused to complete, which shows initiative, but in terms of classes and skills they are so unfocused i'd rather see somebody so focused within a niche with no degree because they knew college would hurt their ambition.

<comment><USER20> Honorary degrees are usually granted to people who have achieved something particularly noteworthy in the field, or who make large donations to the school. I think what you're actually asking for is academic credit for work experience. And indeed some of the lower-tier online colleges such as University of Phoenix or Kaplan University do offer that option. So if you just want to check the box of having a Bachelor's degree then that might be an option, but be aware that many employers don't take such degrees seriously.

</comment></comment></comment></comment><comment><USER21> Huh? That chart in figure 3 tells it all; on the assumption that an employer in the 80s valued a college degree as much as a company in the 2010s, then as more people attain a college degree, it's value (i.e wages) won't move up. In fact you'd expect it to go down but that increase in "productivity" probably offset the downward pressure.

If you asked me, the problem facing workers in their 20s and 30s is that people are having longer careers, either by choice or by necessity. This is coupled with a job market which has never really recovered from the mid 2000s peaks. Until the Baby Boomer generation really begins to retire, the job market will remain tricky to navigate. Compound it all with an ever increasing cost of living (basically, the cost of housing) in pretty much every market and you get the bleak landscape we have now.

</comment><comment><USER22> They state the causes of lower Millennial average pay is an incomplete recovery causing lower participation in the labor market, and lack of unions to force higher wages.

They then go on to say that the Fed should continue depressing interest rates, and that employers should start providing significant parenthood benefits.

I'm not sure with the last two -- continuing to depress capital for businesses and increasing the costs of an employee -- will encourage greater hiring or higher wages.

If it's not profitable to make loans, banks won't provide the capital many businesses need to grow their staff since profits instead have to be more conservatively retained to protect against cash flow fluctuations.

I thought that recent economic research was demonstrating that, when controlled for position and employment type, male and female salaries weren't significantly different. I'm not sure this one makes complete sense as a solution to the problem of employers not hiring people, but I'd definitely be interested to see more analysis on the hypothesis that greater gap coverage for maternity leave could help keep women on track and such, that's particularly cool.

</comment><comment><USER23> Graduates are increasingly less prepared for the working world and burdened with large student loans. It should come as no surprise that higher levels of education have not improved Millennials' standing.

There are more distractions and diversions than ever that prevent youth from otherwise doing something constructive with their time. And universities have been dumbing down curriculum to bring in more (less qualified) students that are buoyed up by federal loans.

</comment><comment><USER24> My empiric opinion is that it's very common among millennials to 'follow their passion'. 'Weakness' that is exploited by business people when negotiating salaries. Mix it with post-scarcity economics and globalization and the phenomenon is explained.

<comment><USER25> Most don't love what they do-they want the world their parents promised would be theirs by going to college.

<comment><USER24> I agree. Older generations made emotional promises that made sense in that moment of History - 'graduate from a good college and the world is yours' \- forgetting that there are ever changing dynamics that easily refute that judgement: e.g. supply and demand. You don't earn a good sallary because you _know a lot_ , you earn a good sallary because the skills that you are able to market have a favorable demand/supply ratio that inflationate their value.

<comment><USER25> Moreover, it was promised (because when people graduated from college in 1973 it made sense) that any college degree was your white collar meal ticket. I still think people who advocate undergrad liberal arts/humanities are delusional or have a vested interest (like they themselves have non STEM degrees)

</comment></comment></comment></comment><comment><USER26> "By all rights, Millennials—people born between 1981 and 1997—should be the highest-paid generation in American history. They are, after all, the most likely to hold a college degree and are working in a period of unsurpassed productivity"

This syllogism is not only unfounded, but basic economics suggests the opposite is true.

<comment><USER27> I have a degree in Economics, and I don't understand what you're trying to say.

There should be a positive correlation between productivity and compensation. How does "basic economics" suggest the _opposite_?

<comment> maze-le I think what he means is: More people hava a college degree. That means more people are in competition to get "white collar" jobs. More competition on the "white collar"-job market, creates incentives to lower the cost per person (There is a bigger share of people willing to be paid less than a generation before). Therefore the average salary per person drops.

<comment><USER28> Also, "more productive" means fewer people are needed to do the same amount of work, so labor prices are being pushed down from the demand side too (as long as the total work being done doesn't keep pace, i.e. low growth).

</comment></comment></comment></comment><comment><USER29> >By all rights, Millennials—people born between 1981 and 1997—should be the highest-paid generation in American history. They are, after all, the most likely to hold a college degree and are working in a period of unsurpassed productivity.

"By rights"? This article goes off the rails right there in the first sentence. It's hard to know where to start with how wrong this is.

If the demand for your skill doesn't change and twice as many people have it, in aggregate the people with that skill are going to make _less_ money. The idea that we're going to educate ourselves into prosperity as a society is daft - if we could all do brain surgery it would be a minimum wage job.

The over-schooling of the millennials is a massive misallocation of resources, fueled by debt, cynical college administrators, weak politicians, credulous bureaucrats, and parents doing the wrong thing for the right reasons. That's not the kind of thing that makes you wealthier. Quite the opposite.

</comment><comment><USER30> That lead photo caption is just horrible.

</comment><comment><USER31> Unemployment ,I graduated in 1981 , newly minted , state certified in welding 1750 cr. hrs. With a lot of hope went for a welding job in Toledo , the line was 100 yds. long . I got up there, the guy shook his head at the guy next to him and said this is the guy we need ; looked at me and said son look behind you all of those guys have families to feed ! Unemployment I could tell you all about unemployment !

</comment><comment><USER32> To hire someone, you need money, and to get money, you need to get hired ... The problem is that there is not enough money!

<comment><USER3> There's plenty of money.

The problem is that spending $10 on capital to make the existing work force more productive is often a better investment than spending $10 on some more labor. It's difficult for labor to argue its importance in such an environment.

<comment><USER32> Being more productive usually let you earn more too.

<comment><USER3> That's why I'm separating capital investment out from labor.

New equipment helps the business earn more, which of course gives them the option to pay their employees more, but if there are lots of people willing to be employees, they business probably doesn't need to pay more.

</comment></comment></comment></comment><comment><USER33> Is it generally accepted that lower federal reserve interest rates increase employment? If so, what's the mechanism?

</comment><comment><USER34> govt subsidies for college degrees have made them the new high school diploma.

</comment><comment><USER35> Boo hoo. Now get back to work! ;)</comment>


Some good javascript tips & handy functions - <AUTHOR> http://code.google.com/p/jslibs/wiki/JavascriptTips

<comment><AUTHOR> Generator expressions:

        [ y for ( y in [5,6,7,8,9] ) ] // is [0,1,2,3,4]

and

        [ y for each ( y in [5,6,7,8,9] ) ] // is [5,6,7,8,9]

Because in for extracts index names, and for each extracts the values.

<comment><USER2> Oh dude, you're the man. I didn't even know you could do comprehensions in JS.

<comment><USER3> Of course it's not cross-browser, but if you're doing server-side JS it might be useful.

<comment><USER4> It's not just server-side or client side these days. Using Rhino I'm using JS for compressing JS files for production and generating documentation for said JS files.

</comment></comment></comment></comment><comment><USER5> Regarding "shuffle the Array":

list = list.sort(function() Math.random() > 0.5 ? 1 : -1);

I'm wondering just how reliably random the result is. Does JavaScript make any guarantee whether the comparison function will only be called once for a given pair of values, and if not, could this affect the result?

<comment><AUTHOR> Well, even if they compare multiple times the order will still be 'random' as in not easily predictable

<comment><USER6> On the contrary, I bet that depending on the exact search algorithm, it would be possible to make some very specific predictions. To pick a simple example, say you use this technique on quicksort of the "pivot on the first element" variety. Given the nature of the algorithm, the first source element will end up near the center of the "randomized" list with high probability, not at all what you'd expect from an unbiased random permutation.

Granted, no real world sort is that simple, and it's even possible that the "clever technique" will produce an unbiased permutation for _some_ sorting algorithms. But why waste time thinking about this when the Knuth shuffle, a proper way to randomly permute a list, only takes a few lines of code more and is so well-documented?

</comment></comment></comment><comment><USER7> Latest Mozilla JavaScript has also Generators (yield keyword), Destructuring assignment (var [a,b,c] = [1,2,3]) and just-in-time compilation.

</comment><comment><USER8> The more I read the less I know</comment>


Why we failed – Good product, not enough waves - <AUTHOR> http://blog.fortrabbit.com/goodbye-webrechnung/

<comment><USER3> These two paragraphs hold a secret :

 _"Our main problem with WebRechnung is that it doesn't fit our standards any more – software, as everything, degrades over time. The interwebs is moving fastly forward, everything is evolving. Adapt or die. We don't want to see this old piece of software (base core is 3 years old now) out there into the wild much longer.

We would love to publish it on GitHub. Unfortunately this software is not really a modern ninja style stand alone application. It is tightly integrated into our monolithic MISH system – mostly written in Perl and impossible to understand for an outsider anyways. And it's really old: dependencies are outdated, large parts of the code are in need of major re-factoring. We would rather rewrite everything before touching this code again."_

One of the challenges of building software on open source frameworks is building _durable_ software. I have been playing with thttpd recently on the Rpi and amazed at how durable this software is. It hasn't been changed a whole lot in years and years, and it just works. And it does that because everything it depends on is also not changing and just works.

The challenge for this new generation of programmers is to create a durable environment where folks could write invoicing software once and have it just work.

</comment><comment><USER4> This highlights something that many start ups fail to properly understand - <USER2> good product is a tiny part of the challenge of succeeding with a business. Without some way of getting the word out there people won't find your product. It doesn't matter how good it is. And if people don't find you, you're dead.

Every start up needs a good understanding of tech to build something saleable, and a good understanding of marketing to bring people to the product. You can't survive without both (caveat: you can buy marketing).

<comment><USER5> You can buy tech too ;-)

Indeed, I'd _suspect_ there are more relatively non technical companies who've focused on the sales and market fit and made a killing on _relatively_ sub-par outsourced tech than vice versa. Could be wrong though!

</comment></comment><comment><USER6> Just because code is old and crappy is no reason not to open-source it. Many successful projects (e.g. firefox) started that way.

</comment><comment><USER7> Please explain what MISH is/stands for.

</comment><comment><USER8> If it's running fine as a free product, why does it have to be end-of-lifed? Does it cost you anything?

<comment><USER9> Sunsetting can also mean no longer having to support it via bugfixes and the such. Time costs plenty when you could spend it building and working on other things.</comment></comment>


Benchmarks of Cassandra, HBase, VoltDB, MySql, Voldemort and Redis - <AUTHOR> http://vldb.org/pvldb/vol5/p1724_tilmannrabl_vldb2012.pdf

<comment><USER3> VoltDB Engineer here. It seems like the authors made some unfortunate choices in the configuration and usage of VoltDB.

A limited number of synchronous loads are always going to present scalability problems and will do a poor job measureing the throughput achievable. Even if it takes 500usecs to round-trip a transaction from the client, that means each synchronous client can only do 2000 per second. To scale linearly to 12 nodes might require hundreds of these synchronous clients.

VoltDB is commonly used with many parallel synchronous clients, such as web front-ends, or asynchronous workloads, such as event feeds. Both of these workload scale very well, so long as there is enough parallelism.

They also used global, consistent, multi-partition transactions for the YCSB scan. I'm not sure if that is necessary, and it would also limit scalability.

I can't speak too directly to the work because the code and configuration doesn't seem to be public. Perhaps I'm mistaken.

It would have been nice if they had reached out to us at VoltDB to verify their configuration and client. They also could have contacted Andy Pavlo at the H-Store who has some experience with YCSB on H-Store.

Still, we can always learn from something like this. One thing we're working on in VoltDB is better performance for workloads that aren't designed to be parallel. This means more performance for synchronous clients and for clients that do global consistent reads. Another area we could improve is more prominent diagnostics to show you why your cluster isn't running faster. That information is all available in our system tables and management tools, but we could probably boil it down to a single status field telling you whether your cluster is starved on client requests, intercluster-transaction-agreement or slow procedures. In our experience, it's usually the first one. I'll go file a ticket now.

<comment><USER4> Hmm, there has to be more to it than that. VoltDB throughput went _down_ scaling from 1 node to 4 in every workload but RSW. This is not a sign of a system that would handle many more requests if only there were more client parallelism.

<comment><USER5> Another Volt developer here.

Short answer, Volt establishes a global order across transactions. Transactions can't be executed immediately because their position in the global order isn't known when they initially arrive. The exchange of ordering information is driven by in-flight transactions. In the absence of sufficient in-flight transactions to drive the ordering process heartbeats are sent every 5 milliseconds resulting in increased latency and lower throughput as the system waits on heartbeats. 2-4x the concurrency should have shown different results.

We're finishing a different transaction initiation system that doesn't produce a global order so that people can do this kind of benchmark and get the expected result. You can switch the beta version on in 2.8.

I also wonder if they had clock skew. The global ordering process is always delayed by the amount of clock skew you have. With NTP properly configured this is 10s of microseconds, but with a typical out of the box NTP config it will be milliseconds unless you have a lot of uptime.

I don't really see this as an excuse and it is why we are eating the pain of a transaction initiation rewrite so people can use Volt the traditional way with small numbers of synchronous client threads.

</comment></comment></comment><comment><USER6> This is like publishing the comparative performances of a Nissan Altima, a Ford F-150, a Honda Odyssey, a Smart Fortwo, a Kawasaki Ninja, and a Gillig Advantage.

Sure, it's possible to determine which one has the best land speed, range, and fuel economy, but you're also sure as heck not going to use a Gillig Advantage to do an F-150's job or vice versa.

</comment><comment><USER7> With these sorts of academic benchmarks I frequently find myself questioning what exactly got tested. The researchers obviously know what they're doing, but that doesn't make them experts on all of the technologies under test. Their description of their HBase configuration in particular leaves me skeptical of how closely it reflects a well-tuned production setup. I think publishing the actual configuration files they used for each technology would be a big help.

<comment><USER8> My thoughts exactly. The HBase numbers point to a "strange" setup.

Disclaimer: HBase developer here.

</comment></comment><comment><USER9> In a system like Redis Cluster (or any other where clients talk with the right nodes directly) it is conceivable that you always see linear scalability as the number of nodes goes up. This fact is not captured by the setup of this benchmark.

If you think at it, in a real-world scenario if you have 100 database nodes, you likely also have not threaded clients, but N different processes running in M different computer systems, querying different nodes independently.

<comment><USER4> Unless you're CPU bound on the client, it's irrelevant whether you're using multiple threads or multiple processes to generate the load. Assuming of course that you're not using a GIL-bound client, which the Java-based YCSB does not.

<comment><USER9> If you look at the Redis graphs there is no linear scalability with N distinct nodes. Now given that Redis has no proxy nor any other node-to-node chat in this setup, how it is possible that it's not linear scalable?

<comment><USER4> I wondered the same thing, but I do know that it's not threads vs processes. :)

</comment></comment></comment></comment><comment><USER10> Interesting, though I miss Riak, Postgres and also a benchmark of degradation effects after running the workload a few 10 or 100 times (due to on-disk / index fragmentation).

</comment><comment><USER11> It's refreshing to see a benchmark that is somewhat thorough and doesn't even make sweeping statements like "X is faster than Y".

Everyone who's ever written a hello world benchmark? Learn from this.

<comment><USER12> Well they did say that Cassandra is better than all the others - <USER2> when compared to HBase - on nearly all measurements (except for high write scenario latencies.)

This makes the decision of Facebook to go with HBase for their new messaging platform back in 2010 all the more strange. Though that was two years ago so things might have changed in Cassandra's favor since then.

<comment><USER13> > This makes the decision of Facebook to go with HBase for their new messaging > platform back in 2010 all the more strange.

Speed isn't everything to a database. AFAIK they chose HBase over Cassandra because of consistency guarantees: eventual consistency is a bad choice for a messaging platform.

<comment><USER14> _eventual consistency is a bad choice for a messaging platform_

Strange you would mention that in the context of Cassandra, since it allows for per-read/write configuration of consistency, from "eventual" to "strong". You get exactly what you ask for with Cassandra, whether its availability or consistency.

AFAIK, HBase only supports strong consistency.

<comment><USER15> This is not the main reason people would choose HBase. For time series data HBase is much better at load balancing the data and at storing the same data in sequential blocks so that in a single operation you can fetch all of the data points that are interesting. Cassandra supports range queries but the last time I saw it wasn't super awesome at load balancing data across nodes when using the OrderedPartitioner. Do I remember wrong?

<comment><USER4> Time series works very well on Cassandra. If anything, avoiding global- ordering hot spots is a Good Thing.

[http://rubyscale.com/blog/2011/03/06/basic-time-series- with-...](http://rubyscale.com/blog/2011/03/06/basic-time-series-with- cassandra/)

[http://www.datastax.com/dev/blog/advanced-time-series- with-c...](http://www.datastax.com/dev/blog/advanced-time-series-with- cassandra)

[http://www.youtube.com/watch?v=F-fYqPu2ciQ&feature=youtu...](http://www.youtube.com/watch?v=F-fYqPu2ciQ&feature=youtu.be)

<comment><USER15> Does Cassandra support Range queries with the RandomPartitioner now?

<comment><USER4> Within a partition, yes.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER16> > Since the Redis cluster version is still in development, we implemented our > own YCSB client using the sharding capabilities of the Java Jedis library.

I just glanced over this, but at this point they might have very well benchmarked driver performance. I also don't see why somebody would compare Cassandra against MySQL and against Redis.

While they all persist application data, their feature sets are so completely different that I don't see for which task people would have to decide between them.

<comment><USER11> I think they wanted to discriminate between them on tasks where someone might reasonably decide for either of them.

<comment><USER16> I haven't seen those usecases yet.

Either you want to scale over multiple systems or you don't. Either you have more data than RAM or you don't.

<comment><USER17> How about Hacker News, and more generally internet discussion forums?

Many discussion forums are SQL backed - but both Reddit and Digg are reported to use Cassandra, a NoSQL system.

<comment><USER16> In this case Redis would be the odd man out. Unless you want to do manual sharding and replication, MySQL also is targeted at a different use-case. I also don't think anybody would run a 40 node mysql master-master cluster

</comment></comment></comment></comment></comment><comment><USER18> Why they didn't tested MongoDB or Riak? Is Voldemort really more relevant than Riak or MongoDB? WTF?!

 _Sorry, boys, but this paper looks like the result of a ill performed research job, done in a hurry of getting published._

</comment><comment><USER19> Another vote for MongoDB

</comment><comment><USER20> Should'ev tested MongoDB too!

<comment><USER21> Wanna mongo measurements too</comment></comment>


Reference Check Job Candidates - <AUTHOR> http://blog.eladgil.com/2013/03/reference-check-candidates.html <comment><USER2> This seems like terrible advice. If someone tried to ask the same question to me about a past colleague multiple times in an attempt to get me to say something bad about them, I'd end the call very quickly.

In fact, the only information I ever give about a past colleague is either that they were great to work with (if their performance was acceptable) or no comment at all (if their performance wasn't). Badmouthing a past colleague is just plain immoral and likely to lead to many closed doors for everyone involved in the future when that past colleague is now heading up a billion dollar company.

You contact references with only one question or you are making a huge mistake. "Did So-and-so work there from X to Y in a position of Z? Was he dismissed for any offense? Thank you for the information, good bye."

<comment><USER3> I completely agree with you. In fact, in certain states/cities, there are legal ramifications for calling businesses or references asking anything outside the realm of 'Did they work there, yes or no?'

</comment></comment><comment><USER4> I do agree that a reference call can be very valuable, but up to 15 minute calls? When you call a reference you are essentially making someone do a favor for someone else, which is fine because the candidate asked for that favor. But I also think it's proper to keep it short and simple so you don't impose too much. It's your job to vet the candidate, not theirs. I used to use a high-level senior VP above me as a reference, and 15 minutes of his time was worth a lot - probably on the order of 50 dollars based on his compensation.

>If possible, find people who have worked with the candidate that the candidate did not suggest you talk to.

 __NO __. This is beyond imposing, it is dangerous and unethical. People talk, gossip travels, and some companies consider employees leaving a very sensitive manner. If I found out a future employer did this to me I would consider retracting my application, even if I had already accepted an offer.

<comment><USER5> Maybe it depends on the industry. But I worked in a field where everyone knew everybody if you'd been around for a few years. So it was an absolute no- brainer to look up who you knew at the companies the candidate worked, pick up the phone and ask what's the word on the street about this guy. And the guy I was calling would typically consider it a favor to me, not to the hire. In fact, failing to call, or to have a network of people who could help on hiring references, purchase references, occasional technical and best-practice questions, would be failing networking 101 and the real no-no at that level. Of course, this would be when an offer was about to be made, and not to anyone who wasn't supposed to know about the hire, or whose discretion couldn't be relied on, and just 1-2 people.

Of course, randomly contacting a bunch of people via LinkedIn would have been bad judgment and a reason for the candidate to question working for someone who would risk putting them in an embarrassing spot.

Also, I don't really agree with the most direct in-your-face questions in the post _unless_ you know the reference _really_ well. If I was the reference those questions would make me uncomfortable and I would shut down the call pretty quick. Would almost make me think the candidate had a buddy testing me for what type of reference I was giving, which is not totally unheard of.

Also, my experience is the check is mostly to check for the red flag, make sure their description of role was accurate, maybe confirm/fill in a gap about something you might not get the full story about from the applicant, how much were they really involved in X, circumstances of departure, basic strengths, weaknesses, would you hire again. If you can't do the technical / cultural fit yourself you're gonna have a bad time.

Other than that, I'm surprised at the pushback... for the most part this is the way it's done.

<comment><USER6> References are not an opportunity to engage in "networking 101." The advice recommended in the post belies a poor screening and interviewing technique.

<comment><USER5> Sorry if I was unclear. I did not mean it was an opportunity to build a network. I meant, at least where I was, if you were a hiring manager and didn't have the ability to tap your network and check through a back channel if someone in the industry was legit, and take advantage of it, you weren't really doing your job.

<comment><USER6> Legit-ness should be sorted out WAY before references are checked.

</comment></comment></comment></comment></comment><comment><USER7> For the record, the fastest way to burn a bridge with me would be to find random people at my last/current company I didn't give you permission to talk to, and go out and talk to them.

If I find out a person at a company I am supposed to work for does this, I won't say anything, I just wouldn't show up my first day or any day in the future.

All relationships start with trust, and while I understand you are trying to protect your business, you are also putting candidates at personal risk while lowering the value of their network.

References are challenging, and I don't plan on relying on them any more than making sure a person can come up with 3 or 4 people that will say something positive about them. I don't care if they are not telling the truth, but it is nice to know that the entire world doesn't hate them.

*In most cases, I have strong bonds with people I currently and previously worked for, and I am highly likely to get an email if you reach out to them. If they don't send me an email, I didn't know them and you won't get good information.

If someone ever calls me and does this, I will first lie to them, then second I will pick up the phone and call the person who they are trying to get references for.

<comment><USER8> This is the piece of "advice" which jumped out at me as being completely horrible.

What if the job search is going out without informing EVERYONE at the prospect's company? What if the prospect hasn't informed his boss that he is looking yet? There is a real chance that talking to some random person at candidate's current job could put their current position in jeopardy. You could also happen to ring up some vindictive asshole who just wants to sabotage the prospect's chances.

</comment></comment><comment><USER9> These are all wonderful suggestions, but personal experience in the US and dealing primarily with candidates from medium to large companies is that reference checks get referred to HR, and the policy is to confirm dates of employment and positions only. Too much litigation risk in the US to say much more.

<comment><USER10> So that's the situation for the US - i wonder how other parts of the world think about that practice, let alone if it's allowed by the law? Is there a list somewhere?

</comment></comment><comment><USER11> This advise seems like it's suited for the business world and not well placed here.

In the tech/startup world, I would advise against doing this sort of in-depth questioning of references. Really, most of these checks are going to be done by your HR team/recruiter and all they're really trying to establish is that they are who they say they are. They're not trying to establish how awesome the candidate is.

The "awesomeness" part is something you need to establish through the rest of the process:

* interviews with the hiring manager and team

* code samples via Github or some other project

* a coding challenge that is reviewed by several people

There are some other things you can do. I for one am a proponent of the "social interview" idea. I like to have candidates come over for some part of the event where they aren't sitting in a conference room and you aren't asking them technical questions. See if you can have a conversation with her, figure out if he's friendly and interesting, decide if you're willing to spend the next two years "in the trenches" with this person.

<comment><USER12> So you're saying that in the startup world, where companies are smaller and each employee matters more, companies should do less work when trying to validate candidates before making offers?

<comment><USER13> I believe he's proposing that startups do more effective validation of candidates, not less work.

The information reported by references (or cold calls) can be vague, misleading, and limited by anti-defamation policies.

On the other hand, information gathered from their publicly shared code, interviews, hanging out, pairing, coding problems, etc. is much more relevant to your question. Will this person fit our startup?

<comment><USER12> He's saying that a particular avenue of information gathering shouldn't be used at all. That's doing less!

Elad is saying that all of those things (interviews, hanging out, looking at shared code, etc etc) should be done and ALSO reference checks should be done.

And of course the information from references can sometimes be vague and misleading. All information about candidates can be vague and misleading. It's the job of a good hiring manager to sort through all of this information and do they best that can to form a complete picture. The more information the better!

</comment></comment></comment></comment><comment><USER14> If I ever found out a potential hiring company went around my list of approved references to find other opinions I would immediately cease negotiations and start warning colleagues about that company.

It's rude. It's invasive. If you are incapable of determining if the person sitting infront of you is a fit for your organization without serious spy work, you shouldn't be sitting in the hiring chair.

References are doing a favor for the applicant. To cold call people randomly trying to dig up dirt is sad and pathetic.

</comment><comment><USER15> I still haven't figured out how to find references who might not accidentally leak the fact that I'm looking elsewhere to the whole office.

I tend to end up putting people from jobs previous to the current one to avoid this, which probably makes the whole process somewhat pointless.

It's all part of the picture when it comes to explaining why the hiring and job search process itself is more of a problem than the availability of hires.

</comment><comment><USER16> I once received a reference-verification call for a previous coworker and friend which started to go in the direction of asking for more than confirmation of employment and general pleasantness to work with, and I wrapped it up as soon as possible with the old "hard stop." It's not my job to pause my _actual_ work to sell you on a candidate. A job reference is not analogous to a letter of recommendation. If my previous coworker was a colossal jerk, I might say, "Hmm, I'm not really sure about that, it was a while ago," and that's it.

</comment><comment><USER17> This is, as the comments suggest, a touchy subject. I generally advise people to never respond to reference check requests with anything other than confirming they were employed.

There is a risk that a bad reference will get you sued by the candidate, and a good reference (where the candidate doesn't work out) gets you sued by the company. Its a lose-lose situation.

That said, one of the really interesting things about the San Francisco Bay area is that there is relatively high job mobility. So a lot of job motion (or not) is from other employees who refer their friends, or suggest that the company pass. Once someone has gone through one of the 'grinders'[1] they get some visibility to other people who worked with them or was aware of them. In my own experience I've met people who I won't work with ever again. And people whom I would work with in a heartbeat. And a bunch of people in between. I'm sure that I land somewhere on that spectrum for a bunch of people in the valley as well.

It is more effective to look and see where a person has been hired and stayed vs hired and bounced out again, to get an understanding of how they fit in or didn't. And if they followed folks from previous jobs or sought out new folks. Was their mobility increasing? (if so they are probably pretty good) or decreasing? (then you might want to probe that, _with the candidate_ )

At the end of the day, digging up gossip on someone by seeking out back channels cannot get you useful data on the candidate.

[1] A 'grinder' is a large 10,000+ employee tech company which is constantly consuming new hires to fill roles due to a relatively high rate of turnover. Examples have been HP, Apple, Google, Sun, Xerox, Cisco, Etc.

<comment><USER18> "a good reference (where the candidate doesn't work out) gets you sued by the company"

Really? Has this ever happened with the plaintiff ending up winning in court?

</comment></comment><comment><USER19> FYI the word is "discreet" not "discrete"

I find that practice (trying to find someone else to give a reference) to be deplorable. Suppose you do talk to a current coworker. If the candidate ultimately doesn't join your company, you've basically messed up future relations between the candidate and the coworker (and possibly the boss -- don't assume that the coworker will remain hush-hush).

</comment><comment><USER20> This is pretty bad advice, sorry.

</comment><comment><USER21> > If possible, find people who have worked with the candidate that the > candidate did not suggest you talk to. You need to do this discretely - e.g. > don't call their current boss to ask how good they are thereby screwing over > the candidate, or alternatively causing their existing employer to make a > big counter offer.

You are not doing a security clearance background check on your future employee. It's one thing if your friend bill worked at company X and you casually ask him about bob when you have a chance encounter one day.

But it's completely unacceptable to engage in this kind of espionage. The reason is that the candidate has not been informed of it, plain and simple. What you're advocating is tantamount to spying. This person might not think it's worth you digging through the coffers of his past just to work at your company. Did you ever stop to think that?

</comment><comment><USER22> Wouldn't it be easier just to ask for their Facebook password?

</comment><comment><USER23> If a recruiter were to read this article and do something like this to me, I would be totally pissed.

If I were to find out that a recruiter took the time to find extra references by rummaging around my social networks and calling people I didn't mention, I would feel some serious serious mistrust. This is the kind of thing that would actually make me reject an offer from a company.

If you have found that this technique actually works for you, then what you are actually discovering in reality is that you probably: 1) Have an inadequate technical interview process. 2) Have inadequate tech recruiters.

</comment><comment><USER24> "If possible, find people who have worked with the candidate that the candidate did not suggest you talk to."

Would be nice if the article suggested how you go about finding these people in a fair and responsible way.

<comment><USER25> Private Eye. I hear Jack Nicholson's real good.

</comment></comment><comment><USER25> 5 references seems a bit overbearing...

</comment><comment><USER26> This is just plain bad advice.

</comment><comment><USER27> This is good advice, and the first comments don't do this helpful article justice. I have just been reading about reference checks in standard reference books about hiring (as part of updating my differing FAQ about hiring procedures before I post it to my personal website), so I can vouch for the correctness and helpfulness of what is said in the submitted article. The procedures mentioned are legal, and you are a chump if you don't use them.

Recycling some electrons, here is an adaptation of a comment I made on Hacker News about a year ago:

There is an art to checking references. Even if a company has a policy of giving bare minimum information, find out a TELEPHONE NUMBER of someone in that company who knows your candidate and start a conversation. I was given a specific script of questions to ask back in the 1990s when I was a community volunteer for my local public school district, doing reference checks on superintendent candidates. A consultant advised the school district (and through the district, me) on how to do this. If you talk to someone directly by voice, and have a good list of specific questions to ask about the candidate, you will be AMAZED at what people say, policy or no policy. Company policies don't keep people from sharing stories with curious listeners. The key is to learn what questions are legal to ask and reveal the most interesting stories about the person you are thinking of hiring. There are consultants who can advise you about checking references, and, as people who have had actual hiring authority for important jobs will say, checking references is a lot less expensive than making a wrong hiring decision. Once you've learned the questions, you know what to ask.

I've just asked my consultant Google, and he suggests several sets of useful questions to ask when checking references:

[http://hiring.monster.com/hr/hr-best-practices/recruiting- hi...](http://hiring.monster.com/hr/hr-best-practices/recruiting-hiring- advice/job-screening-techniques/reference-checking-questions.aspx)

<http://www.drgnyc.com/list_serve/Jan24_2005.htm>

[http://www.acadweb.wwu.edu/hr/Employment/InfoForHiringOffici...](http://www.acadweb.wwu.edu/hr/Employment/InfoForHiringOfficials/HiringPersonnel/Exempt/TelephoneReferenceChecks.pdf)

[http://www.best-job-interview.com/reference-check- questions....](http://www.best-job-interview.com/reference-check- questions.html)

<http://www.k-state.edu/hr/employment/referencecheck.htm>

[http://pbsbo.ucsc.edu/personnel_payroll/staff/recruit/ref_ch...](http://pbsbo.ucsc.edu/personnel_payroll/staff/recruit/ref_check.html)

[http://www.bridgespan.org/Publications-and-Tools/Hiring- Nonp...](http://www.bridgespan.org/Publications-and-Tools/Hiring-Nonprofit- Leaders/HiringTools/Reference-Check-More-Than-Formality.aspx)

[http://jobsearch.about.com/od/referencesrecommendations/a/re...](http://jobsearch.about.com/od/referencesrecommendations/a/refercheck.htm)

AFTER EDIT: My comment on the initial response to this comment of mine is that I am on record

<https://news.ycombinator.com/item?id=5227923>

as saying here on HN that if you want to hire a good worker, you should give the job applicant a work-sample test. Research tells us that, and wishful thinking can't make that untrue.

But if you are relying on anything said implicitly by someone's references to decide whom to hire, you had better check the references. People lie on their reference lists. They do that a lot in industries where the custom is not to check references. (We have read some stories about that here on HN over the years.) If your work-sample procedure for hiring is airtight enough, checking references may not add much to your hiring procedure, but it is legally safer to check references (to avoid employer legal exposure for negligent hiring) than to not check references. You prove you are a gentleman and a good sport by letting the honest people be discovered as honest when you check their references. Engaging in wishful thinking on this issue protects you neither legally nor practically.

<comment><USER28> _Even if a company has a policy of giving bare minimum information, find out a TELEPHONE NUMBER of someone in that company who knows your candidate and start a conversation._

So, now you're going to get someone who is dumb enough to respond to what could conceivably be a social engineering call, and asking them for personal information about a third party.

I can see no way that this will end up with useful and/or actionable information.

<comment><USER27> _I can see no way that this will end up with useful and/or actionable information._

You haven't actually done this. As I mentioned in my parent comment to your comment, I actually have. That was very helpful to the public school district where I was serving as a community volunteer for knowing which superintendent candidate to hire. By the way, the personnel director of our public school district reported to the school board last year that when teachers leave the employment of the district, he almost NEVER hears reference check telephone calls, which he would be happy to answer, from any subsequent employer of those teachers. He thinks that is seriously weird. Don't people want to be careful when they hire teachers of young children?

<comment><USER29> Yes, that's why they have criminal background checks performed and are licensed by the state.

The risk of a defamation lawsuit is a very serious and real thing.

</comment></comment></comment></comment><comment><USER30> I'd flag this, but the OP is getting lots of deserved hate and that would stop the pile-on process, so I won't.

Thanks, HN.</comment>