Keybase is not softer than TOFU - <AUTHOR> https://keybase.io/blog/chat-apps-softer-than-tofu <comment><USER3> "How often do resets happen? Answer: if you're using WhatsApp or Signal, all the freaking time.

With those apps, you throw away the crypto and just start trusting the server: (1) whenever you switch to a new phone; (2) whenever any partner switches to a new phone; (3) when you factory-reset a phone; (4) when any partner factory- resets a phone, (5) whenever you uninstall and reinstall the app, or (6) when any partner uninstalls and reinstalls. If you have just dozens of contacts, resets will affect you every few days."

I guess I don't have "dozens of contacts", but getting a new phone/resetting a phone isn't really that common of a thing in my circle. I feel like for the average user, they wouldn't do this with their phone more than like once every year or two. So I guess if you have like 600 people you talk to on these apps regularly then that works out to daily, but for me at least this isn't that big of a deal and was pretty much understood from the outset.

<comment><USER4> This isn't really the issue. From further down on the article:

""" There's a very effective attack here. Let's say Eve wants to break into Alice and Bob's existing conversation, and can get in the middle between them. Alice and Bob have been in contact for years, having long ago TOFU'ed.

Eve simply makes it look to Alice that Bob bought a new phone:

Bob (Eve): Hey Hey

Alice: Yo Bob! Looks like you got new safety numbers.

Bob (Eve): Yeah, I got the iPhone XS, nice phone, I'm really happy with it. Let's exchange safety numbers at RWC 2020. Hey - do you have Caroline's current address? Gonna surprise her while I'm in SF.

Alice: Bad call, Android 4 life! Yeah 555 Cozy Street.

So to call most E2E chat systems TOFU is far too generous. It's more like TADA — Trust After Device Additions.1 This is a real, not articifical, problem, as it it creates an opportunity for malicious introductions into pre-existing conversation. Unlike real TOFU...by the time someone is interested in your TOFU conversation, they can't break in. With TADA, they can. """

The quote you linked is relevant because it means that you can't simply ignore this problem; resets are _fairly_ common, common enough that you can't just delete the key-loser's account (for example). However it doesn't have anything to do with the actual security flaw (if we want to call it that, it's really more of a UX / messaging problem) being discussed.

<comment><USER5> That exchange is not realistic, it's easy to ID family or friends by how they conversate. At worst you can ask a challenge question. The prompt that a device has changed is perfect to heighten senses for a quick ID. I disagree that physical contact is necessary as TFA (and industry lore) seem to recommend:

> You must now reestablish identity, and in almost all cases, this means > meeting in person and comparing "safety numbers" with every last one of your > contacts

Perhaps the alert could be a little more alerty and in red and read something along the lines of "Hey! Your buddy's safety code has changed, make sure they sound normal and aren't acting weird and creepy asking for information"

<comment><USER6> How many people are realistically going to pester their friends & relatives with "challenge questions"? I bet even the majority of folks in the HN crowd don't/wouldn't.

<comment><USER5> I'm not saying you have to know their first pet's name. I'm saying it's usually pretty obvious through regular conversation if someone is who they say they are, and worst case, if you're suspicious, you can ask about some shared past event without them knowing that they're being challenged.

<comment><USER7> You are basically saying that social engineering never works, but there are many stories about how social engineering _sometimes_ works.

<comment><USER8> Why does it have to _never_ work? No technique is perfect.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> Much of the criticism of how gently WhatsApp and Signal handle key resets misses the mark. Widespread adoption of end-to-end encrypted messaging is an effective countermeasure to passive collection and blanket surveillance. In order to get that widespread adoption, you can't be showing people skull-and- crossbones warnings every time they swap out a SIM card.

Speaking from my experience getting journalists and political campaigns set up with signal in 2017-2018, the early scary key change warnings were off-putting to people and made them reluctant to continue with the messenger. At the time, I was in contact with 100-150 people via signal and quickly ran out of patience with anyone who insisted on a safety number check. But the UI at the time encouraged that level of paranoia.

I continue to believe that making key changes as painless as possible for users is the correct approach as long as there are ways to harden this behavior in the settings, for the benefit of the far smaller set of people to whom MITM attacks are a credible threat.

<comment><USER10> I agree with your assessment about key change management. With that said, I do like the device history trail that Keybase uses. Keybase has a better multi- device story - in that it has a multi-device story at all. I understand what they're trying to do preserving message history - I do prefer my conversations to be ephemeral by default.

I'm OK with the compromise that Signal has made with key management -- there are people that I really care to have private communication with, and people who I prefer to have private communication with. I verify the former, and don't bother with the latter unless we happen to be bored together in the same room.

On a side note, thanks for your efforts over the last two years!

<comment><USER9> The user base for Keybase and Signal is so different that it makes sense to have such different behaviors.

Thanks very much for the kind words!

<comment><USER11> I would have thought that these two groups would have more in common than not.

Who do you think constitutes these disparate user bases and how are their perceived expectations different?

</comment></comment></comment></comment><comment><USER12> Matrix handles this by exposing the device keys to the user so they can make decisions about whether to trust new devices (and I believe identity key changes mean you wouldn't be in your rooms anymore -- but in order to change identity keys you would have to delete you entire Matrix account on the homeserver).

If a new device has shown up, your messages will be blocked from being sent until you verify the new device. To be fair, it is too easy to blaze past the warning -- and it can happen often in large rooms. As a result, it's a little bit cumbersome at the moment, but with device cross-signing coming down the pipe and the new verification system (which is much better than Signal's IMHO -- you just check both devices have the same string of 7 emoji on their screen) it's getting a _lot_ better.

<comment><USER13> I still think Keybase is right — multi-device or some kind of multi-trust model is best so the key revocations aren't happening so often. I remember this problem with PGP and most people did not take the key verification seriously.

And the problem with SSH that was pointed out in the article is funny now because of cloud services where servers are constantly being destroyed, so keys are changing frequently, unless you save and persist that private key for the server in your configuration. Which I've realized a lot of companies are simply not doing, which leads to people straight up ignoring key verifications in their ssh config.

<comment><USER14> SSH is even worse in many corporate environments, because you also have to cross an SSH gateway to transit into from the corp fabric into the prod fabric.

At AWS this meant we had to hop to a different gateway for each region, and then onto the (stateless) production host which was recreated from scratch on every deployment... There's no way a human was going to keep all those keys straight, and everyone just disabled SSH host verification.

<comment><USER15> This actually seems like the right use-case for SSHFP records. They allow for storing the SSH key fingerprint in a DNSSEC secured zone.

Normally this is considered rather weak, because of the reliance on DNSSEC. However, in this case I think its a very easy to take step that'll be miles better than just trusting everything.

The stronger alternative is Certificate-based SSH, but that is a _lot_ of hassle. As it requires setting up a CA, handling revocation, and actually distributing the certificates.

<comment><USER16> It depends on how transient your servers are, but you can get pretty good mileage out of a checked in known_hosts file.

</comment></comment></comment></comment></comment><comment><USER17> I love Keybase.

Actually I tried to install the app a couple of days ago.

But there is no version on F-Droid and the version on Play Store has Firebase Analytics baked in

Is there a plan for a clean F-Droid version?

<comment><USER18> They also have no (as far as I could find) way of installing Keybase without root permissions on Linux. I tried looking for a way to install keybase without "sudo dpkg -i keybase.deb" but had no luck. In the end, since the people I'm working with use it, I had to spin up a VM to install it in so that keybase wouldn't mess up my Debian installation.

<comment><USER19> Keybase developer here.

Keybase uses root privileges only for making the magic /keybase directory available, where you can access your KBFS files (the redirector allows different users on the same system to see their own files). Keybase and KBFS run as unprivileged daemons (via the systemd user manager where available).

As giancarlostoro mentioned, you can unpack the .deb file and run the binaries out of there. If you put the binaries in your $PATH, you can even symlink the systemd unit files to your ~/.config/systemd/user and use the systemd user manager to manage your custom Keybase install. Note that the KBFS mount will not be accessible at /keybase, but instead at another location writable by your user (see [https://keybase.io/docs/linux-user-guide#configuring- kbfs](https://keybase.io/docs/linux-user-guide#configuring-kbfs)).

Finally, our build script at [https://github.com/keybase/client/blob/master/packaging/linu...](https://github.com/keybase/client/blob/master/packaging/linux/build_binaries.sh) builds all components and assets without root and lets you handle the packaging and install after that.

If you have questions or more custom use cases, feel free to join the keybasefriends team to talk about it or make a GitHub issue!

<comment><USER18> Thank you for your your detailed response! It would be really nice to have this information in the official documentation.

<comment><USER19> Added: [https://keybase.io/docs/linux-user-guide#installing- keybase-...](https://keybase.io/docs/linux-user-guide#installing-keybase- without-root-privileges).

Let me know (I'm the Keybase user modalduality) if you have anything else to add or any questions. Cheers!

</comment></comment></comment></comment></comment><comment><USER20> (A very tiny fwiw): you /can/ create a backup in signal and use it to transfer seamlessly to a new device, without triggering new safety number checks. The user flow sucks, but it is possible.

<comment><USER21> Backups are only possible on Android, and the most recent Signal non-beta builds have had this feature broken, making backups you had useless.

Unless your willing to open a bug and be a pest for 2 weeks (and still have access to your old phone/leave it registered) I wouldn't plan on retaining your messages or keys across phones. Its a huge weak point of Signal.

<comment><USER22> Even in that case, you could only backup on device storage, had to write down (!) a huge set of numbers and manually copy files around. The whole flow is awfully terrible to any user - at least WhatsApp can sync your profile over Google Drive.

Last time my phone died, I lost all of my Signal group memberships, because the app is incapable of transfering those to a new phone without backup... and it's also incapable of doing the backup automatically. Those UX choices continiously baffle me - it's like authors didn't learn anything from the failure of PGP.

<comment><USER21> Backups are low priority for Signal, and they don't want to make security compromising choices like WhatsApp does with backing up.

It'd still be nice if the UX and reliability of restoring from backups could be improved.

</comment></comment></comment></comment><comment><USER23> Requirements to use phone number, let alone, as an identifier is major complaints I have for many of messaging apps. It really limits usable cases as I have plenty of people I would love to interact but not necessarily want provide my phone numbers.

I love Keybase for this aspect, but something I don't like about it is its device name handling. They don't allow decommissioning old device names, so I end up having 'MyLaptop' 'MyLaptop 1' 'MyLaptop 2'...

<comment><AUTHOR> Oh interesting. I don't think we've talked about this decision publicly, so I can write about it for a second. Not letting people re-use a device name is an inconvenience, I admit, but arguably it's not like other cryptography inconveniences, where people are confused, troubled, etc. We figured people would say "huh, weird requirement" and pick a different name and move on.

The goal is a 1-1 mapping between devices (keys) and these names. So whenever we need our UX to talk about a key, it can talk, safely, about it in terms of device names. Once committed to your chain of signatures, "Laptop-Warhol" means a specific device key, and it can't be used again. So, for example, if one of your Keybase installs wants to tell you "oh, Laptop-Warhol just added a new device, iPhone-Vangogh" then it doesn't need to look like this: "Key 34858234589234895897234598734 added key 90123845890230948234234324."

If Laptop-Warhold could mean multiple devices (keys), well then we'd need to start talking about the keys. Which is a nightmare for usability.

A lot of this decision was driven by something we've seen with apple devices. Every now and then I'd get a popup on my computer - say when updating iOS - that said something like "you just started using iMessage on a new device, 'chris's iphone'. if you don't know what this you should freak your shit out." well - it has basically said that so many times with the same names over again, that I can safely assume that it's a near-useless warning.

Note I mean unique to you; 2 different users on keybase can name their devices the same.

Generally speaking...it's been a goal from the beginning that names on keybase are meaningful. Similarly if you look up "chris" in in our merkle tree (which is pinned to bitcoin) that leads to a deterministic chain of signatures. inside that chain, where I mention "work-imac-warhol", you're guaranteed to see the same answer as I am. So "chris" is as good as a key fingerprint or safety number. And so is my device name.

<comment><USER23> I understand where you are coming from, but I my irritation comes from the fact that they device cannot be revoked from my account and recycle the name. (e.g. OS reinstalls, etc. I don't really like naming my device MyMainPhone-2, logically because it's not my second phone. It's a same device to me.) If the device that's _active_ already have that name, I would agree with your decision that duplication is not allowed.

Maybe some people are more irritated by that than others (I'm certainly former-type), perhaps consider "advanced" to remove existing names for those who knows what they are doing?

<comment><USER24> You're not the only one annoyed by this.

</comment></comment></comment></comment><comment><USER25> Let's not forget what Signal and the Signal Protocol (used by WhatsApp) have achieved: making end-to-end encrypted chat EASY and accessible for the masses, for many of whom "security" is password123. It's important in our post-Snowden world.

<comment><USER26> I'm not sure why this is downvoted. The reason WA has so much popularity is that it is easy to use and you could use wifi. Signal is not as easy to use (less features) but more secure and so privacy conscious people like it. But people don't like switching to Signal because "it is hard". Making a more secure app is good, but we have to question "do we want people using pretty good e2e or do we want to make the perfect app first?"

<comment><USER27> I don't think your question is as easy to answer as you might be implying. I certainly don't know the answer. My main concern is: what is the downside to "pretty good e2e"? Without understanding what that means, we can't evaluate the situation.

"Good" is better than "best" if "best" is not available seems obvious, but it certainly isn't true in a lot of cases. If I tell you that X secure and you trust it to be secure, when it actually has problems -- that might be worse than me telling you that X is not secure.

People are bad at evaluating risk. If I want to pass notes in class and don't want my teacher to know what the note says if I get caught, then rot-13 is probably "good enough". But if I'm a whistle-blower for a government agency, my security needs are quite a bit higher. We can never make a perfect app, but I'm not sure I could define what "good enough" looks like for the general populous. It's completely reasonable to me that different groups have different opinions on the matter -- and I think that's a good thing.

<comment><USER26> > If I tell you that X secure and ...

Is Signal or WA in that regime? Literally the only reason I don't use WA is because it is owned by FB. But as far as I'm aware both are cryptographically secure. Yes, I'm aware WA has a metadata problem.

So what's good enough? I'd say that if you need a state actor to crack it, I'll call it good enough. At least for the general populous. Any more difficulty that can be made is a bonus imo.

Clearly we can't get a perfect e2e app. So at what point in time do we say "we also need other features that people want so that they'll use our app". That doesn't mean "stop working on encryption" (you can never stop that) but "we're at a good enough point to start targeting a larger market." I think something like Signal is there. Stop focusing on the security geeks and bring in the general public.

</comment></comment></comment></comment><comment><USER28> The thing I find about this article the nicest is that they published the audit result document.

[https://keybase.io/docs- assets/blog/NCC_Group_Keybase_KB2018...](https://keybase.io/docs- assets/blog/NCC_Group_Keybase_KB2018_Public_Report_2019-02-27_v1.3.pdf)

</comment><comment><USER29> The argument is that people don't bother to keep their verified connections up to date. But come on, how often to MITM attacks happen? For those rare cases where people are doing stuff important enough that it becomes a possibility, I would guess the security conscious individuals would become more diligent.

For the rest of us, it seems that doing it on occasion is still worth it. As I understand, Signal is designed to never indicate over the wire who has checked safety numbers. Thus, a MITM anywhere on the network creates a risk of becoming discovered, which is a cost in itself.

<comment><USER30> What's the point of using an encrypted chat at all if you don't care if it's hacked?

<comment><USER31> A big benefit of end-to-end encryption is that it makes it impossible for the service provider to suddenly start doing silent mass surveillance of their userbase. A solution doesn't need to be 100% perfect to achieve this goal.

Somehow, nearly every article on the subject completely misses this, and instead keeps moving goalposts on reasonable endpoint security.

<comment><USER32> Exactly. I only use signal for chat, and I tell my family and friends that they must install it if they want to send me text messages. It's not because I fear being hacked, it's because I don't want highly personal conversations about sex, drugs, chronic illness, and skeletons in the family closet getting picked up by global passive surveillance.

<comment><USER33> Yep. And Signal is as good as it gets for end users with zero technical knowledge. I've had many family members and friends install it on their own and not skip a beat.

</comment></comment></comment></comment></comment><comment><USER34> Show us the server: [https://github.com/keybase/client/issues/6374](https://github.com/keybase/client/issues/6374)

<comment><AUTHOR> It _seems_ you (someone from the Signal project?) are actively diverting from the point, with what is ultimately a security theater request. Keybase's app - which IS open source - doesn't trust the server at all. We could be running anything server-side, regardless of what we do or don't publish. Meanwhile, Signal's story is "you MUST trust our server, over and over again," as the blog post explains. Unfortunately there's no way to know what's happening on the server. So being like Signal and publishing your server source is strictly worse than being like Keybase and not (yet?) publishing server-side source. At any time, Signal could be throwing in these fake key upgrades, either due to running other source code on purpose, or being forced to, or just plain getting hacked. The most malicious Keybase server could not.

This comment may be of interest (we could release server code at some point, and I will take this as a vote), but I hope people reading this aren't distracted by Signal's flaw here.

[edit: chilled a bit!]

<comment><USER35> _> But to be clear, you're actively diverting from the point.

...

I get why you showed up here, but you're really not addressing the point of the post at all, and in fact you're trying to distract with the suggestion that Signal's publication of that code protects people from this flaw. It doesn't. At all._

Wow, that's a pretty hostile (and accusatory) response to a fair ask. This is one (small) step removed from accusing someone of shilling/astroturfing.

Let your product stand on its own merits. If you have a good reason why you won't open source Keybase's server implementation, own it. Don't undermine requests to open source the code by publicly accusing people of supporting a competing product.

The person you're replying to didn't make an argument in favor of Signal - <USER2> any other competing product, for that matter. In my opinion, _your_ response is actively distracting from _their_ request.

<comment><USER36> Even if they publish their server code there is no way for anyone to verify that it's the code they are actually running and it would be just a PR move. If the client implementation is good there should be no way that the server can compromise any message.

<comment><USER37> It's a step toward people running their own servers, either federated with Keybase proper, or just as a personal instance. That would be valuable for quite a number of enthusiasts. Federation (like email/XMPP) is a very reasonable feature for any forward-looking communication platform.

<comment><USER38> Then this is not a request for transparency but for them to change their business plan

</comment></comment></comment></comment></comment></comment><comment><USER39> I'm wondering why the article fails to mention that there is a sufficiently good and easy mechanism to compare and verify the new safety numbers. You just talk to your peer and read the numbers - and the peer can verify them.

This will fail when AI software gets really good at imitating voice in real- time during casual talk, but we're not there yet (or - if that is my threat model, I'll find an out of band way to verify)

<comment><USER40> A sufficiently good mechanism from a cryptographic standpoint, but which from a usability perspective totally falls down because people never do it.

It's a very worthy goal to make these events rare and scary so that users might actually bother reading those numbers out loud to each other.

</comment></comment><comment><USER41> I'm not a security guy, but wouldn't the most seamless approach be to encrypt the key collection with a master password and store the encrypted key collection on the server? So on a new device you'd download the encrypted key collection and then decrypt it locally?

If they forget their password, they can re-upload it from a validated device with a new master password.

<comment><USER42> Biggest issue is single point of failure for total access to all devices. Get\guess\beat out the master password and its game over on _all_ connected devices.

<comment><USER43> Not only that but this also enables offline attacking of the password. If you can compromise the Keybase server and grab the encrypted passwords, you can then attack it at your leisure with whatever computing power you can scrounge up, over whatever time duration you want. And when you break it, as long as any of the included devices are still on the account, you'd have complete access to everything.

Requiring existing devices to be actively involved in provisioning a new device prevents all of this.

<comment><USER41> Ah, that makes more sense.

So in Keybase, what does device to device provisioning look like? "Hey, you've just set up this device - a message has been sent to all your other devices, OK the message and come back here and you'll be good to go"

<comment><USER44> You just scan a QR code. Or if you want to do an offline device you can type in a a set of words that represent the key.

</comment></comment></comment></comment></comment><comment><USER45> I actually _do_ reverify safety numbers out-of-band every time they change.

<comment><USER46> I would actually like to see some numbers or a survey on how many people actually do this. Anecdotally I don't think I know a single person who ever verifies safety numbers out of band. Of the approximately 50 people I use to talk on signal with regularly not including large group chats of people I don't know as well, not a single person has ever tried to reverify me nor have I tried to reverify any of them.

Would be cool to see what the numbers are for reverifications.

<comment><USER45> I normally don't do this in-person, but some kind of out of the loop approach is something i do every time.

One time i got a postcard from a friend with their safety number on it and nothing else. I recognized their handwriting to complete the loop.

<comment><USER47> Would you mind elaborating?

What you described sounded interesting

</comment></comment></comment></comment><comment><USER48> Why can't we just have government certificate authorities for the average Joe?

Ultimately, while people may have very little trust for the government in general, the one thing that people do trust the government for is establishing identity. We use government ID papers to establish our right to work, to open bank accounts, to enter legal agreements, and to cross borders. Why should communications be any different? We don't need to trust the government with the _content_ of the communications (and we shouldn't), by not providing the government with the private keys. But why can't I get the government to sign a public key for me?

The issue it raises is whether people will eventually get locked out of society if the government decides to get antagonistic with somebody by revoking their public key and refusing to issue a new one, given a society where such a scheme is popular. But we don't have any sort of such protection today - the government can seize your passport, seize your driving license, freeze your bank accounts. A society in which the government solves identity issues for the digital age is only a net improvement over the status quo.

<comment><USER49> Because not everyone wants to communicate under a singular identity.

</comment></comment><comment><USER50> > _Similarly, in SSH, if a remote host 's key changes, it doesn't "just work," > it gets downright belligerent:_

Funny enough, I have ranted to friends/coworkers about sysadmins completely replacing machines and not telling anyone. How do I know it happens? BECAUSE OF THIS EXACT WARNING.

</comment><comment><USER51> If there was an iPad app that works in landscape mode I'd be able to switch most of my groups from other apps to keybase.

</comment><comment><USER52> Smart stuff and fun:

"Did you though, or did you just scroll down here?"

</comment><comment><USER53> If a single party in a chat has their keys reset that's not a MITM attack because a MITM would need to rekey to both parties. If the two clients communicate via at least one uncompromised service to communicate that their counterparty's keys have been reset, they might be able to detect a MITM

Keybase's example Cozy Street is not a MITM attack (i.e. an attacker has inserted themselves between two parties and can get the plaintext), it's just impersonation.

If it was a real MITM attack both Alice and Bob would get rekey notifications, unless they both confirm whenever they get a rekey notification a MITM attack is possible.

I also think that crypto is stuck in the early 90s by thinking real world meetups are the only way to authenticate keys. If you know what your chat friend sounds and looks like, willing to submit a video, and don't think your adversary can fake such a proof, a simple video of you reading your pubkey/safety number is sufficient. Is that scalably practical? no but it is possible.

That said: keybase is doing cool novel work, I commend them for advancing the state of the art.

</comment><comment><USER54> I don't know how much weight such warnings will hold, given how well we know people ignore cookie warnings and the rest...

<comment><USER55> Look at the number of non technical end users who will determinedly download .EXE files, or run them from their mail client, and click through all of the Windows 10 "do you really want to run this untrusted software?" warnings in order to successfully install cryptolocker type malware on their computers.

If you give people a way to click "yes/accept/run" and they are determined to accomplish what they think is their intended task, they will just blow through any warnings.

<comment><USER56> You don't even need to observe the average end user. Just look at software developers aka "technical experts" using npm, NuGet, Maven, and all the other package managers. Digital signatures? Nope, just run the code on your machine, please. Bonus points for allowing code execution in user context to "configure" the package and placing executables in $PATH.

npm here being exceptionally secretive on what it will install as dependencies as it can reach tens of thousands packages very quickly.

</comment></comment></comment><comment><USER57> Actually, I think it is wrong to call it TOFU as it simply doesn't require the user to opt-in to anything. Instead, it seems more like the thing the XMPP people call 'blind trust before verification' [1]. I am not quite sure if it is exactly the same as 'blind trust before verification' changes its behavior as soon as you explicitly verified the keys.

That way everybody can use somehow e2e encrypted messages, but if you really care about the security you validate your keys and get a real trusted e2e encryption.

[1] [https://gultsch.de/trust.html](https://gultsch.de/trust.html)

</comment><comment><USER58> I've been using keybase for quite some time now and I absolutely love it. I suggest everyone give it a seriously try and check out some of the popular public teams chats before finalizing your opinion of it. I'm glad I did.

<comment><USER59> Best thing for me besides chat working with some friends regularly is the amazing file sharing option between multiple parties easily.

</comment></comment><comment><USER30> Keybase's UI is still bad. It prominently highlights the "Let them (the eavesdropper) in" button, and the warning has so much red that it's hard to read the text.

</comment><comment><USER60> Their solution seems reasonable, but why not allow export of an (encrypted) private key for use on other devices?

<comment><USER43> Keybase allows you (nay, encourages you) to set up a "paper key", which is a private key that you store offline (e.g. print it out and stick it in a safe). You can then use this paper key to provision new devices. This way if you lose all of your devices simultaneously (or just have one device to begin with) you don't need to go through an account reset to add a new device.

Note that even with this, any new device added using the paper key still generates its own independent private key. Having per-device private keys is important to be able to revoke devices, and to be able to track which devices were responsible for any given action.

<comment><USER60> To quote Kanye Omari West[0], "I like this."

[0][https://youtu.be/PsO6ZnUZI0g?t=95](https://youtu.be/PsO6ZnUZI0g?t=95)

</comment></comment></comment><comment><USER61> I can never get keybase to address this article about them:

"Keybase, we have a problem." [https://freehuman.fr/posts/20238f40da8b01357816236af097d2ae](https://freehuman.fr/posts/20238f40da8b01357816236af097d2ae)

Maybe user /kbModalduality can

<comment><USER62> A happy KB user, let's bring that article down:

1\. A _server_ runs KBFS which syncs data with the server. No one could stop you from disabling it on startup and running wherever you need it.

2\. Any app running without AppArmour/SELinux or outside of network namespace could get your real address. Latter is relatively easy to set up, I'm running VPN by default and all the apps inside are running only in the namespace with VPN tunnel device.

3\. Last time I checked I could make my own package.

4\. I don't have time to check it atm.

5\. Link for audit results is somewhere in the comments on this post.

6\. Works with uBlock on.

7\. Some paranoia rant without looking how KB works: it's push/pull mode, all messages are stored as files you could sync.

8\. Based on previous points.

9\. Each connection is under TLS. Plaintext messages could be read via RAM, and if someone could read it, KB would be the last problem.

10\. You could make your own.

11\. Fixed. _#ls -l /keybase_ would show you the symlink KBFS_NOT_RUNNING to /dev/null, but shows correct directories under user running the app.

12\. Hello GDPR.

13\. Like everyone in the world does.

14\. Bad outro.

</comment></comment><comment><USER63> I like it! They should apply this to the problem of TOFU in HSTS. For many HTTPS sites it's trivial to hijack them by getting a user to switch to a different device, as the devices' apps often don't synchronize HSTS databases.

</comment><comment><USER64> Keybase is awesome.

I wish they would take my money as a customer. I don't want to see them to away.

</comment><comment><USER65> Key rotation is a hard problem. One idea is to host the public key in a txt domain record. For example yourname.com that you can also use for your email and blog.

</comment><comment><USER35> There are a few claims made here which I'd like to see clarified. In the spirit of transparency, let me declare upfront that I use Signal but I am not affiliated with them, and I don't really have a dog in the race here. I'll reference the published NCC security review[1] for this comment. Overall I'm happy to see a published cryptographic review of this protocol.

First, under "The Full Security Picture" heading in this article, it's claimed that forward secrecy is supported via time-based exploding messages. Pages 19 and 20 of the NCC report explain that, "The default chat protocol does not allow for forward secrecy since the same keys can retain indefinitely on a users device." So forward secrecy is not assured by default under this chat protocol, is that correct?

The NCC report goes on to say that, "Exploding messages introduce mechanisms for message deletion and forward secrecy; however, it is not clear to the user that keys and messages could remain on their device beyond the period specified during message creation." I interpret this to mean that there _is_ a way to assure forward secrecy - which is exploding messages - but you're not making that explicit in this announcement. This seems a little disingenuous to me because in your FAQ, the first answer criticizes Whatsapp for compromising forward secrecy using the backup feature, but you don't have forward secrecy enabled by default in your chat protocol.

Likewise, this announcement makes a point of mentioning how other apps require you to trust the server due to resets, and why trusting the server is bad. But page 20 of the NCC report explains that, "While the default Chat encryption protocol does provide for message confidentiality and integrity, it does not provide for security in the face of device and server compromise, as keys and ciphertext are stored for a potentially indefinite period of time." So is it correct to say that unless users specifically enable exploding messages for their conversation (which is not the default), they actually _do_ need to trust the server?

There are also a few drawbacks to the ephemeral messaging scheme NCC found that I think should be explicitly disclosed, because they don't require too much technical detail:

1\. There is no deniable authentication on the default chat protocol. While exploding messages provide deniable authentication, this property fails in a group with more than 100 participants. It's fair to question whether that's a realistic place to expect deniable authentication, but it should probably be called out.

2\. Exploding messages are based on the local client's system clock. Therefore it's possible for an exploding message to be indefinitely retained on another device by e.g. manipulating the local time.

___________________

1\. [https://keybase.io/docs- assets/blog/NCC_Group_Keybase_KB2018...](https://keybase.io/docs- assets/blog/NCC_Group_Keybase_KB2018_Public_Report_2019-02-27_v1.3.pdf)

<comment><USER66> Keybase affiliate here.

Correct: forward secrecy isn't on by default. We think there's a trade-off here. With forward secrecy, your old messages won't be visible on a new device, but users want this since Slack (and others) make it seem natural. However, you can opt-in to forward security on a per-message or per- conversation basis.

The report says "device _and_ server compromise." Decryption keys never leave the user's client. What they mean is if: (1) the server's stored data is compromised; (2) your phone is also compromised; and (3) the messages weren't marked ephemeral; then the attacker might be able to read past messages, even if the user tried to delete them (i.e., did Keybase really delete the ciphertexts?). This line of reasoning is correct and one of the primary motivations for key ratchets. I don't think the report is claiming that users need to trust Keybase's server in general. They do need to trust Keybase to delete messages that are marked deleted, which would mitigate the attack above if conditions 1 through 3 are met.

<comment><USER35> Thank you for clarifying that, especially the second point. Looks like I was misunderstanding the report then.

<comment><USER66> Thank _you_ for taking the time to read the report!

</comment></comment></comment></comment><comment><USER67> Having encrypted backups means that you are throwing away PFS.

</comment><comment><USER68> Kind of OT but the belligerent SSH message example they used is good comedy

</comment><comment><USER69> Use threema. It has this problem (and the phone number one) solved.

<comment><USER70> I would argue that the benefit of solving this problem mentioned is far outweighed by the large downsides of using proprietary software.

</comment></comment><comment><AUTHOR> I don't know the moderation policy on title changes at HN, but I just changed the title of the post. Internally at Keybase - and thanks to a conversation with a peer - we've been feeling pretty guilty about calling out a specific project that we think is basically the gold standard outside Keybase.

We'd rather focus on the positive solution to the problem (which Keybase has implemented), rather than just pointing a giant finger at any other services which have the problem we're trying to address. I think I personally will sleep better tonight this way.

Still we want this conversation to continue.

<comment><USER71> Sleep well! We've updated the headline here.

<comment><AUTHOR> <3

</comment></comment></comment><comment><USER72> In an ideal world, I would love to use a really secure app for communication. However, my choices are limited when most of my friends are on WhatsApp or Signal. That is just the unfortunate reality of the way social networks work.

<comment><USER9> Both WhatsApp and Signal are secure apps for communication, and your friends are right to use them.

</comment></comment><comment><USER73> Very reasonable criticism. I'm now happy for never having used Signal.</comment>


Ask HN: Mall School – take a break from your normal M-F routine and teach - <AUTHOR> I've been teaching programming at the mall for a few months now.
Idea: compete with big schools like:
https://generalassemb.ly
By using the mall (free space) and hiring teachers that work for free.
i.e. there are lots of programmers like me earning big salaries that would love to take a break from their normal M-F routine and teach one or two days a week.
But trying to schedule students and find serious students is a lot of work. I'm emailing people all the time and often they don't show up.
Mall School will charge the students a small fee (so they take it seriously) and have an admin handle all the scheduling. Teachers just show up at the mall with their laptop and a student is there waiting for them with their laptop.
The Mall has:
Bathrooms, wifi, table with power outlets, food court
Feedback welcome. Where do you think this idea falls apart if it does?
https://pbs.twimg.com/media/CzkhVIHUUAAAi9Y.jpg
Picture all across USA at every mall, 1000's of people learning programming and 1000's of teachers getting a lot of satisfaction teaching and helping new people get into this industry. <comment><USER2> I think it's going to fall apart when the mall realizes you're running a small business out of their food court.

You might also find that some parents would resist letting their kids go to the mall to meet some people with computers.

Maybe try to find a local high school(s) that are trying to get a CS program off the ground? A high school would have bathrooms, wifi, power, and maybe even an actual computer lab. You could schedule coding lectures after school on certain days of the week. You'd probably need something like a faculty sponsor though.

<comment><AUTHOR> well look at this table:

[http://higher.team/assets/6000_office-8f65a06e12b1ed7007b8c5...](http://higher.team/assets/6000_office-8f65a06e12b1ed7007b8c540528af8d4525f0a220b1ae27ee213ba1aa20b0e7e.jpg)

it's saying "please come to the mall with your laptop and work here" and I always buy something at the mall on the days I'm here. I don't think the mall would have a problem with it as long as it's not 100s of people crowding the table all at once. Just two people with laptops looks normal not "running a business".</comment></comment>


iPad jailbreak after less than 24 hours - <AUTHOR> http://appadvice.com/appnn/2010/04/ipad-jailbreaked-day/ <comment><USER2> Isn't that the method which was already known on iPhone but not published, exactly because they were waiting for iPad to come out? (so they could use the same method on both)

<comment><USER3> Yes. And it's userland, meaning you can visit a website in MobileSafari and jailbreak the device, similar to jailbreakme.com back in the day.

<comment><USER4> does this mean that any website can potentially run code as root on my iPad?

<comment><USER5> If that's the case then yes, yes it does, so Apple will fix this like they did when that was the case back with the iPhone.

</comment></comment></comment></comment><comment><USER6> Hooray! The children are saved!

<comment><USER7> I don't think the children were ever in any danger. If anything, having to jailbreak the iPad is more of a challenge and promotes the hacking spirit. Further on down the thread someone mentions the fact that in Turkey, most of the iPhone users, even non-power users, have jailbroken iPhones. I think that in this respect Apple's attempt at security may even promote hacking and jailbreaking. It may be a healthy experience to start with a locked device and have to jailbreak it.

<comment><USER8> It's the only reason I've ever been interested in Apple's mobile devices. I joined the iPhone Dev Team right after the iPhone came out and worked on it through the initial unlock. Once the challenge was gone, I moved on to greener pastures. It wasn't to get some specific goal (I never really installed any apps on my phone), but just to do it and have some fun. Everyone should reverse-engineer something like that at some point -- the feeling can't be beat.

</comment></comment></comment><comment><USER9> I don't see why companies even bother doing this, if it'll just get jail broken a few days after release....especially if it's such a popular product like the iPad.

<comment><USER10> Regular, non-power users won't jailbreak.

<comment><USER11> Does that justify spending (presumably) tons of money and time on trying to prevent power users from jailbreaking?

<comment><USER12> I doubt Apple is spending much money to stop jailbreakers. They just don't make it easy. In most cases these jailbreak methods are exploiting unrelated bugs to get root access on the phone. Stuff Apple has to fix. For example: The 2.x Safari jailbreak method. If you look at other closed platforms like gaming consoles you can see some serious anti-cracking efforts. The Xbox 360 for example has a very sophisticated security system. Microsoft scans devices for known exploits and even bans them from Xbox Live for life if found. Apple could make iTunes check the device for modifications. They could force the device to constantly report back to Apple and report a modified device. They could ban the device from accessing the App Store. All things considered Apple isn't going overboard to stop jailbreaking. I would guess it's just folded into the iPhone OS security team's normal day to day work.

<comment><USER13> The biggest proponent I see for Apple wanting to stop jailbreaking is app piracy, which I heard was a big deal, especially around the decent $10 games from companies you've heard of (EA, etc.).

Personally I'm on the side of the fence that would rather have no homebrew or jailbreak if it prevents piracy, but I'm sure others would disagree with me.

</comment></comment></comment></comment></comment><comment><USER14> This isn't so surprising to me. The iPad runs iPhone OS and the 3.2 seed has been available to developers for some time.

"iPhone OS" is a bit of a misnomer, since it refers to the software running on iPhone, iPod touch, and now iPad. (<http://en.wikipedia.org/wiki/IPhone_OS>)

<comment><USER15> It is surprising, because the 3.2 Seed only ran in the simulator, which is _completely_ different than a real device.

</comment></comment><comment><USER16> Slightly OT but is anyone aware of variant OS installations on iPad (was the mentioned Cydia one example?).

Love the feel of device, would be much happier browsing with Chrome, and copying media/software as I wish. The Apple hardware/software coupling always frustrates me.

<comment><USER17> Cydia's a package manager, aiding in software installation. First the software has to be available, though.

I haven't heard of serious attempts to support another OS on the iPhone or iPod Touch, even. Linux on the iPod fizzled out years ago.

</comment></comment><comment><USER18> Good; so there is at least a glimmer of possibility that the iPad can be useful for something.

</comment><comment><USER19> This is why I F __*ING love hackers.</comment>


Feds Hike Interest Rate to 0.75% - <AUTHOR> http://www.cbc.ca/news/business/federal-reserve-interest-rate-1.3896402 <comment><USER2> It's not the "feds" like the federal government -- it's the "Fed" which is short for Federal Reserve Bank. The Fed is not part of the government. It's as "federal" as Federal Express. [https://www.youtube.com/watch?v=1pVV4n2lKHk](https://www.youtube.com/watch?v=1pVV4n2lKHk)

<comment><USER3> > It's as "federal" as Federal Express.

That's not really correct. The Federal Reserve's Board of Governors are appointed by the President and confirmed by the Senate. More than half of the members of its Federal Open Market Committee come from the Board of Governors (and so are Presidential appointees).

The Federal Reserve is not a normal government agency, but neither is it a non-government entity like Federal Express. It is something in between.

<comment><USER4> The libertarian and gold bug stuff about the Fed actually being a secret private bank drives me wild. I think it's because the shriven government does so little, in terms of producing and not redistribution compared to other nations.

I mean there's all kinds of arms length government agencies. There's state broadcasters like the CBC or BBC with similar independent board of governors/directors. The Illinois Department of Financial and Professional Regulation's Real Estate Disciplinary and Enforcement Boards is something like half real estate professionals as provided by law. There's sewage districts and stuff that nobody knows about with taxation powers and independent election of leaders are they private? It's such a weird meme to have such persistence, especially among techies.

</comment></comment></comment><comment><USER5> Reminds me of "I survived the Fed rate hike of 2015" [http://www.redbubble.com/people/redct/works/19325459-fed- rat...](http://www.redbubble.com/people/redct/works/19325459-fed-rate-hike)

</comment><comment><USER6> It is Interesting that Interest Rate hike gets no interest on HN.

</comment><comment><USER7> Concerns me that the rate hike is primarily based on KPI reporting that less people are claiming unemployment. Oh well, guess you have to hike the rate to lower it again.</comment>


Internet architects propose encrypting all the world's web traffic - <AUTHOR> http://www.wired.co.uk/news/archive/2013-11/15/encrypting-all-web-traffic <comment><USER3> I know this may be a bit off-topic, but since we're discussing HTTP/2 already, I can't help but wonder. The recent attacks (BEAST, CRIME, etc.) have all relied on being able to build an oracle from (sent) body content and uncover header elements. The attacks only get better in time, so I expect to see even more attacks like this.

If we're going to overhaul the HTTP spec in any case and go for framed messaging - why not go for separate header/body compression AND separate encryption keys? As far as I understand, that would block the whole family of attacks. Separate compression contexts would prevent the attacker from building an oracle from body to attack the header. Separate keys and encryption contexts should prevent padding and mode oracles against the header.

I know crypto is hard, and I _know_ there would be devilishly tricky details to figure out. So I have to assume someone has already thought of this, has had it vetted out for flaws and then discarded the idea.

But if so, why? Apart from requiring at least twice the key size in negotiation (compute oversized shared key, split for header/body and likely HMAC keys), are there any other obvious or non-obvious technical reasons not to do this?

<comment><USER4> I think the focus on the dichotomy between the header and the body is misguided. If you separate them then an attacker can't attack the header using the body, but that's just inviting the attacker to find a way to insert data into the header to attack other header fields, or find secrets in the body that shouldn't be revealed.

Also, it seems like you could get the same benefit as using a separate key and encryption context by just padding the header to a block boundary.

The real problem with these attacks is that they aren't attacks on the cipher at all. You could be using a one time pad and the attacks would still be effective. What they really are is an attack on content-revealing efficiency. The only defense is to prevent the length of the message from depending on the existence of matches between attacker-supplied plaintext and secrets, and doing that in a way that allows non-cryptographer web developers to not screw it up basically comes down this: You have to disable TLS protocol compression. It exists at the wrong layer of abstraction because the TLS layer doesn't know what data is acceptable to compress against what other data.

But you still want to use compression, so the compression has to exist at a higher level in the stack, i.e. in HTTP or even HTML, where the web developer can specify exactly which data has come from an untrustworthy source so that it won't be compressed against secrets.

</comment></comment><comment><USER5> One thing I've really thought is consistently screwed up with security proposals is _what do you do when it might be compromised_?

Users invariably click through the big red web page because they still need to use their email at the end of the day, and that page provides no information as to what's going on.

If the certificate doesn't verify, don't give up - try a different route to it (sadly we've taken away source-routing for other security reasons). If that fails then start a Tor session and try connecting through that.

Then show me a little diagram that shows where we think the problem is so I can think about what might be the problem.

<comment><USER6> The problem is the false positive rate. It's like strict parents and car alarms. Car alarms go off without good reason all the time, they're ubiquitous and thus almost completely useless. When was the last time you heard a car alarm and investigated whether a car theft was in progress?

Similarly, a parent which tells their kids that pot will kill them instantly, prevent teenagers from watching PG movies, and so on is more likely to have their advice completely ignored when there is an opportunity to do so. And then you end up with kids who get into oxycodone, binge drink, and have unprotected sex.

The strictness of browser security is pretty ridiculous and not very helpful to the end user. If I go to google news on my phone I get half a dozen messages about certificate mismatches, which I have to dismiss individually. It's a dumb system with even worst UI. Except the UI is the foundation on which a sense of trust or alarm is built.

</comment></comment><comment><USER7> I would like to see in the HTTP/2 spec allow http traffic, with the requirement that http 2 traffic must be signed. This would allow publicly accessible resources (including javascript) to be cacheable, without compromising security. Random thoughts:

* The signage could be done in headers (HTTP/1.0 compatible)

* Works with all existing internet caches that don't modify pages they cache

* (some proxies inject headers into pages, so the spec should be resistance to this, and throws out all unsigned headers)

* Signature must cover cache expiry and full page url (including FQDN, port number etc)

* don't send cookies, user-agent or other identifying details over http 2 (helps with caching, and privacy)

* Could have a requirement that all https 2 traffic can only link to resources on https or http 2

* If the signature in the http 2 message fails, browser could fall back to https 2

* A redirect could sign the content it is redirecting to (to securely load resources from 3rd party CDN)

Edit: formatting

</comment><comment><USER8> Lots of discussion from two days ago: [https://news.ycombinator.com/item?id=6724292](https://news.ycombinator.com/item?id=6724292)

</comment><comment><USER9> Is it folly to continue with https given that the certificate authority tree of trust has been compromised?

Maybe baby steps are a good option, but I'd like to eventually see all traffic signed with keys stored in a distributed ledger like namecoin. Good on the IETF for moving forward though.

<comment><USER10> There are degrees of brokenness with SSL.

Even if certificate authorities are compromised by major governments, SSL is still enough for now to keep my credit card safe from the shady guy in the back corner of the coffee shop. Or to keep my Google searches private from the techies at my ISP.

There are some upcoming workaround that will help too like cert pinning or DANE.

</comment></comment><comment><USER11> Which is all well and good until a three letter agency compromises the issuing CA, which in turn leads to a false sense of security.

<comment> grey-area That's really a separate issue though.

Yes, we also need to tackle the crazy mess of certificate authorities and their monopoly on trust from browsers, but that doesn't mean encryption is not worthwhile.

<comment><USER5> Yes it does.

It has been _trivial_ to bolt encryption to things for a long time now. Encryption is _worthless_ without trust. Ubiquitous encryption more so because it's no longer a surprise when something is encrypted, it's just expected that you have your snooping client MitM the traffic with dummy certificates.

It doesn't stop 3-letter agencies (though really, people should stop pretending that's where the real threat lies for most people) and it doesn't stop informed hackers or intrusive corporate firewalls.

<comment><USER12> Intrusive corporate firewalls MitM SSL sessions because the client is already compromised (IT installs the firewall's cert as a trusted CA). _Any_ solution can't route around a situation where you don't trust the client machine.

<comment><USER5> Right but that's my point: it doesn't matter if it's encrypted. It matters if it's encrypted AND you've established trust.

I'd go so far as to say that trust is actually more important - <USER2> someone can read my messages is less important then verifying they're what the sender _intended_ to send.

<comment> grey-area I agree it's really important, but we need both parts of the puzzle, not just one. Out of interest, what would your proposal for a new trust model for server communication be?

<comment><USER5> Web-of-trust rather then CA-rooted, with more attention placed on who the signers are.

i.e. if I'm using my bank, then what matters to me is whether the bank is certified by my government, not VeriSign or whoever. If it's a foreign bank, then it matters whether their government trusts them, and it matters to me whether MY government trusts them.

If it's NOT a bank, then maybe my trust requirements are different. But the UI for all this and details are everything - but I think we've definitely stuck way too much import on that little green padlock icon without doing enough to educate users moment to moment about what it means.

</comment></comment></comment></comment></comment></comment></comment><comment><USER13> What, if any, affect might this have on net neutrality? I imagine that it would make discrimination by carriers more challenging.

<comment><USER14> As soon as a service becomes relatively popular, the carriers will know about it, and throttle it manually until they "pay up".

</comment></comment><comment><USER15> I feel that while it's good that we're encrypting HTTP more and more, there is too much focus on it, and other protocols are being neglected. Most SMTP and instant message traffic, for example, is not encrypted and authenticated.

</comment><comment><USER16>
       > HTTP/2 to only be used with https:// URIs on the "open" internet. http:// URIs would continue to use HTTP/1 (and of course it would still be possible for older HTTP/1 clients to still interoperate with https:// URIs).

Seems like the most viable option, and leaves open the option of later implementing TLS Relaxed.

</comment><comment><USER17> Given the proposal to encrypt everything, I wonder if the architects are going with standard SSL or the extended validation SSL. Will we need to provide full business details and such just to get HTTPS on our websites, or will a simple credit card payment complete the process?

</comment><comment><USER18> What about IPsec?

<comment><USER19> I agree, it's not necessarily the right OSI layer we should be trying to secure.

Unfortunately, NSA sabotaged IPv6 goal of "encrypting everything by default":

[https://en.wikipedia.org/wiki/IPsec#Alleged_NSA_interference](https://en.wikipedia.org/wiki/IPsec#Alleged_NSA_interference)

Specifically,

"IPsec was originally required in IPv6 before RFC 6434 made it only a recommendation"

<comment><USER20> Given the current state of IPsec/IKE, making it "required" is pointless. AFAIK there is no standard for opportunistic session setup, so random hosts don't know how to speak IPsec to each other even if they both implement it.

</comment></comment></comment><comment><USER21> With CAs, or without CAs?

<comment><USER20> The proposal to encrypt everything doesn't require CAs (but of course it doesn't show the lock either).</comment></comment>


What do you think of our new online magazine for Millennials? - <AUTHOR> http://www.ygenmag.com <comment><AUTHOR> The idea is to create a simpler, more curated version of Thrillist for Millennials living in major cities. This is our MVP - eventually we'll build a crowdsourced content platform to try to innovate on how lifestyle content is created, discovered, and marketed.

Open to any feedback, willingness to get involved, or thoughts on the site so far. Thanks!</comment>


GC Tuning Confessions of a Performance Engineer - <AUTHOR> http://www.slideshare.net/MonicaBeckwith/gc-confessions <comment><USER2> Having worked with all sorts of GCs in the past, I basically stopped using them altogether some years ago in favor of manual memory management. I have to say, it's been liberating and so much easier to write performant code since I know what memory I need instead of relying on the computer to guess (and giving the computer hints as to how to guess). I have no doubt that GC research and systems is making good forward progress, but I can't imagine myself using them again.

<comment><USER3> I love GCs. They give you better memory throughput (in exchange for more footprint and higher latency, although latency can be made rather low), and they let you build and use very scalable concurrent data structures. On large machines with lots of cores and lots of RAM, they let you work with large, in- memory data sets very efficiently.

<comment><USER2> You will never beat a tuned system without GC with a GC system because of all the instructions necessary to traverse references and such. Having lots of cores and lots of RAM just means you can eat the cost more easily. It just means you can afford to let the VM do the work instead of the programmer.

Also, no matter how much RAM you have, cache sizes are more or less the same, and cache line misses hurt.

<comment><USER3> > You will never beat a tuned system without GC with a GC system because of > all the instructions necessary to traverse references and such.

That's not at all how it works. The generational hypothesis means that most objects die young. Allocating them is a simple, uncontended pointer bump in the thread-local allocation buffer (as fast as stack allocation), and freeing them is free, as they are never traversed. They are much, much (much) faster than malloc/free. They do however increase the frequency of young-gen so they have an indirect cost.

When it comes to throughput (i.e. total time the application spends doing memory management), modern GCs handily beat malloc/free. What you can do, however, with manual memory management is all sorts of arena allocations, but then you have to be careful when sharing pointers (Rust helps with that). Then there's the question of concurrent data structures, that are very, very hard to do well without a GC.

> cache line misses hurt.

What does that have to do with GCs? If anything, copying GCs bring related objects together, so the prefetcher can help. What affects cache misses (in Java) is the lack of arrays-of-structs, which is scheduled to be fixed, with the addition of value types, in Java 10.

<comment><USER4> I really dislike the comparisons of TLAB to stack allocation. Stack, by its nature, is going to be hot in cache. TLAB, once filled up, will be retired and possibly assigned to a different thread. But even if it's not assigned, it's constantly moving forwards, and not revisiting the same space. You'd need prefetch to be perfect, and then on top of that, you'd need to make sure that by the time you go to allocate again, the prefetched cachelines have not been evicted.

Languages/platforms with a GC should still use, support, and encourage stack allocation for temporary memory -- this _is_ your TLAB!

There is a cost to traversing references; card marking and generational collectors just reduce the amount of references you need to visit, but it doesn't mean reference chasing isn't requiring extra instructions. Finally, don't forget that card marking requires write barriers, which is extra instructions (and possible cache misses) on each reference store (modulo trivial ones, such as new allocations, where JIT knows it's not required).

<comment><USER3> While everything you say is 100% true, it is also a second-order effect, with a much lower magnitude than the primary GC performance behavior for short- lived objects.

Obviously stack allocation is preferable to TLAB allocation (and there's no reason to allocate objects with stack scope on the heap), if only for the fact that it never triggers a collection. Nevertheless, Java allocation/collection of short-lived object is much closer in cost to stack allocation than to malloc/free.

<comment><USER4> I'm not sure memory locality effects can be considered secondary, unless their effects are completely dwarfed by something else the app is doing (e.g. there's no point in discussing this topic for i/o bound workloads).

I don't think malloc/free should enter this conversation because languages that use malloc/free do so very infrequently, and for the cases where dynamic memory needs to be allocated frequently, they use specialized memory managers within the application. This is also subject to which allocator is used and what the application's allocation pattern is. There're suboptimal GC mechanics as well in some cases, such as CMS tenured space using free chunk lists with no compaction, so any young GC that requires promotion can possibly increase the young GC time because the GC needs to find appropriate free block size, and if there's fragmentation, this may take a while. Point being is that malloc/free vs GC isn't quite as clear cut on its own, nevermind that malloc/free aren't called that often. Generally speaking, though, if you can give GC ample headroom in terms of RAM, it'll have better throughput than incessant malloc/free use (which, I argue, is rare in properly written applications).

<comment><USER3> > I'm not sure memory locality effects can be considered secondary

What is secondary isn't the TLAB/stack performance ratio, but that ratio vs malloc/TLAB.

Also, I'm not sure why you think locality matters much here in the case of stack reuse. Within each frame, the stores always come first, and those go in the store buffer (and the reads are from the store buffer, too), so those are pretty benign cache misses.

> which, I argue, is rare in properly written applications

Sure, it is rare in "well written applications", but how costly is it to write a well-written application in a large team, and how much extra performance can you get? Remember, we're not talking about a DSP, a microcontroller or a net router when we're discussing GCs, but big, complex applications. Nobody is claiming you can't beat a GC given enough work (though it's harder the more concurrency is involved).

Also, think about what kind of data we're talking about. The interesting data is database data, and that has both arbitrary lifetime as well as concurrent read/write. And if you don't use malloc/free, at best you need to write your own manual memory allocator which is just as complex, and at worst you basically need to write your own GC.

<comment><USER4> Ok, we keep talking about malloc -- which malloc impl are you specifically referring to? There are many allocators out there these days, so let's be a bit more concrete. If not specific name, at least the class of allocator. Most of the common ones you'll find support thread-local allocation buffers, for starters.

>Sure, it is rare in "well written applications", but how costly is it to write a well-written application in a large team, and how much extra performance can you get?

>Also, think about what kind of data we're talking about. The interesting data is database data, and that has both arbitrary lifetime as well as concurrent read/write. And if you don't use malloc/free, at best you need to write your own manual memory allocator which is just as complex, and at worst you basically need to write your own GC.

If we're going to talk about databases, then "well-written" better be one of the top concerns, and team size should be irrelevant to that. And the more mechanically sympathetic of a product you're building (and db's are right up there in pretty much all aspects: cpu, i/o, net, mem, etc), the more you need to have control over those resources.

Have you, for example, looked at how postgresql manages memory? sqlite? redis? memcached? nginx? varnish? And, as I mentioned in the other reply, most of the big data java solutions end up rolling their own off-heap mem management infra using the same techniques as you'd use without GC.

<comment><USER3> > Most of the common ones you'll find support thread-local allocation buffers, > for starters.

And what about concurrent deallocation?

> Have you, for example, looked at how postgresql manages memory? sqlite? > redis? memcached?

Not too well (basically lots and lots of locking, much of it is very coarse- grained). Our spatial in-memory Java database (SpaceBase) offers an order-of- magnitude better performance in concurrent usage (and much better scalability with core number). We do over 200K transactions (with lots of contention) per second on a single 4-core laptop without breaking a sweat (concurrently with the application itself), and over a million on a large server (with some careful tuning).

But even for less super-concurrent databases, C++ databases don't outperform Java ones. In this benchmark, the Java databases (H2 and HSQLDB) almost always outperform MySQL and Postgres: [http://www.h2database.com/html/performance.html](http://www.h2database.com/html/performance.html) (and I don't even know how the Java solutions handle concurrency, whether they do locking, optimistic locking or a clever combination, like SpaceBase).

In both cases, the amount of effort put into the Java solutions is orders of magnitude less than the C/C++ solutions.

> nginx? varnish?

Those are (virtually) read-only use cases. That is very easy to do concurrently no matter how you manage concurrency. The trick is concurrent writes, not reads.

<comment><USER5> > But even for less super-concurrent databases, C++ databases don't outperform > Java ones. In this benchmark, the Java databases (H2 and HSQLDB) almost > always outperform MySQL and Postgres: > [http://www.h2database.com/html/performance.html](http://www.h2database.com/html/performance.html) > (and I don't even know how the Java solutions handle concurrency, whether > they do locking, optimistic locking or a clever combination, like > SpaceBase).

It's hardly surprising that a vendor's own benchmark shows it as winning against the competition.

<comment><USER3> Let's suppose for a second that the numbers are biased in the Java databases' favor. I doubt that other benchmarks would show such dramatically different results that would have those databases losing by much. So maybe they're not ahead as those number show (though I have little reason to doubt them), but a little behind. The difference would still not justify the claim that C++ provides superior performance for databases.

Also, it's an open source database with no commercial support by the authors, so I wouldn't really call them "vendors".

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> The slides are interesting enough that I went looking to see if the actual talk that goes with them was online anywhere; found it at

[http://chariotsolutions.com/screencast/philly- ete-2015-7-mon...](http://chariotsolutions.com/screencast/philly- ete-2015-7-monica-beckwith-gc-tuning-confessions-of-a-performance-engineer/)

</comment><comment><USER7> For some reason, GC is on of those areas that I have zero desire to learn sufficiently to tune. That's probably why OP has a successful career as a consultant.

When making tradeoffs in development, a simpler GC with predictable behavior is worth losing a lot of raw performance to me. Thus I find myself drawn to platforms like erlang/elixir or Haskell where GC is isolated.

I suppose the rational approach is build on Hotspot and when it becomes a problem hire a consultant for instant performance gains (assuming low hanging fruit is in abundance).

<comment><USER3> In 80% of the cases, simple GC ergonomics (i.e. let HotSpot figure out the tuning) are more than ok, and in 95% of the cases, trivial tuning is enough.

The thing with Erlang is that the GC doesn't work on any shared memory data structure (like ETS), so you pretty much have to delegate any shared data to an out-of-process database, even in simple cases that are easily addressed with ConcurrentHashMap, ConcurrentSkipListMap/Set, or CopyOnWriteArrayList/Set on the JVM.

<comment><USER7> I generally use an Agent in Elixir for anything shared and there's very little cognitive overhead (though perf does take a hit). Agent.cast/2 is helpful for low latency modifications (at the cost of losing back pressure).

As stated, it's a tradeoff.

<comment><USER3> But, if I'm not mistaken, an agent isn't concurrent. You make your shared data a bottleneck which is exactly what concurrent data strctcures are meant to prevent.

</comment></comment></comment></comment><comment><USER8> Interesting article. What I see is another failure of industry to solve the root of the problem: different GC strategies work for different applications and should be selectable. Might even allow pluggable GC's. This is the path taken in JX Operating System and certain academic works on improving Java runtime. JX's first tier of memory management just gives a certain amount of resources a component/process/app can use. The second tier is its GC, which can be different per application. So, you could partition your application into components with associated GC's that focused on throughput, low-latency, or productivity. Then the problem shifts to making an optimal scheduler for the app and GC runtimes. There's already good schedulers, strategies for building them, and even tools for automated planning of scheduling strategies. Any of these might be integrated to optimize that aspect.

Overall, using pluggable garbage collectors, feeding a planner some constraints, and compiling the result sounds much easier that this person's day job. Embedded Java already does some of this while some systems did it in hardware to avoid most issues entirely. Enterprise Java should adopt such a method. Meanwhile, reading of these nightmares, I'll continue avoiding such GC-based tools wherever possible in my work.

JX Operating System for reference [http://www4.cs.fau.de/Projects/JX/publications/jx- usenix.pdf](http://www4.cs.fau.de/Projects/JX/publications/jx-usenix.pdf)

</comment><comment><USER9> I wonder, isn't it working on the stack much faster than working on the heap anyway?

So when you query data on the heap, shouldn't you just query larger objects to put it on the stack, and work from there, instead of using the heap so often ?

With that in mind wouldn't that render garbage collecting almost irrelevant if your code is well designed, by not working too much on the heap ?

It's true that more ram makes the GC more relevant, but if it's an excuse for negligent software design, maybe GCs are not such a good idea. It's good to have features that make the job of the programmer easier, but if it only save the time of the skilled programmers who knows a little about how it works underneath, is it such a good idea ?

You can hardly convince that simple tools and predictable behaviors in machines are not the safest way of having fair results.

A programming language is already a big shortcut to work faster, and I doubt you should try to work even faster if it means creating new drawbacks.

<comment><USER10> The stack and the heap are both just regions of memory, so there's nothing inherently faster about using the stack. The difference is the memory management strategy: Usually, stack variables are all allocated at the beginning of a function, and all deallocated at the end. If all of the memory your program needs is tied to a particular scope like that, then it's certainly faster than any other memory-management strategy. If you need to share data between scopes, build complicated data structures of indeterminate compile-time size, or allocate large blocks of memory (the stack is usually limited in size), then you have to do something different (like use the heap).

That said, I think the Rust approach to memory management is really interesting: try to tie all allocated memory to some scope, and keep track of when ownership is borrowed by or moved to a different scope.

<comment><USER4> The inherent benefit to the stack is the memory region stays hot in cache due to natural use of the stack. That's your fast reusable buffer for temporaries.

In addition, of course, cleanup/reclaim of the stack space is pointer bump, so you get pointer bump allocation and deallocation, effectively.

<comment><USER10> True, but you could do the same thing in any region of memory. The only thing that using "the stack" buys you is that there are a few special instructions to allocate / deallocate one machine word at a time (if you store the "top-of- heap" pointer in a specific register), and you get a bit more locality by virtue of return addresses being stored next to your locals and temporaries. (For x86 processors at least; maybe other architectures treat the stack in a more special way, I don't know.)

<comment><USER4> > True, but you could do the same thing in any region of memory. The only > thing that using "the stack" buys you is that there are a few special > instructions to allocate / deallocate one machine word at a time (if you > store the "top-of-heap" pointer in a specific register)

This is pretty much what a TLAB is in Hotspot JVM, except of course the pointer never moves backwards (once the TLAB is filled up, it retires, and then can get assigned to a different thread to start allocating from the beginning). Each allocation into the TLAB moves allocations further into the region -- there's never any reuse until the TLAB starts afresh.

The stack locality comes from the rest of the stack execution mechanics keeping this region very warm, and the majority of the read/write action to the stack is localized (apart from large stack allocations that may temporarily expand the region's use).

<comment><USER11> Except that a TLAB is worse for cache than a stack.

Look at what happens when you call a bunch of small methods that each allocate a couple things. With a stack, everything stays in nice proximity automaticially. With a TLAB, you quickly fill the TLAB, get another one, get another one, and so on.

Effectively: the stack does any garbage collection that can be statically determined for free, keeping data in cache that actually matters, whereas with TLABs you need to wait for it to pass through GC before any of it is reused.

<comment><USER4> Right, that's why it bothers me when TLAB and stack are compared.

</comment></comment></comment></comment></comment></comment></comment><comment><USER12> Performance is nice, but correctness is more important. Complex programs written without any GC tend to have use-after-free and double-free vulnerabilities. Programs written using automatic reference counting only (without a way to catch cycles) tend to have memory leaks. If GC is killing performance, then that is a failure of the specific GC or programming language design. Don't blame the applications programmers for wanting GC semantics; that's a completely reasonable thing to want and something that language designers are capable of providing.

In the specific case of Java, I think the real problem is not having value objects (like C#'s struct) or generics with non-reference types. Serious GC problems happen not because GC is too hard, but because in Java an ArrayList<int> of a million items makes a million tiny objects. This problem is entirely Java-specific.

<comment><USER13> Specifically about ArrayList<Integer> \-- there are specifically a bunch of projects (Goldman Sachs Collections, trove, FastUtil, Koloboke) that solve the array of primitives problem.

I would say that's a Java-standard-libary-specific problem.

<comment><USER11> And then you have to do a copy just to pass it around to any other external libraries. And you lose the advantages of generics.

You end up with 9 copies of everything, that are 99% the same except for a couple find-replaces. If not more. (For instance, if you have a method that takes two generic arrays, you need 81 copies! Even if they are the same type you still need 36 (!) copies.)

Not a good solution.

<comment><USER13> How do you end up with 9 copies of everything? I'm not sure I follow that.

If you are passing in massive ArrayList<Integer> lists to external libraries, you need to re-evaluate what you're doing. What will that external library do with this?

<comment><USER11> Look at java.util.Arrays for a concrete example of the problem:

        static int 	binarySearch(byte[] a, byte key)
        static int 	binarySearch(char[] a, char key)
        static int 	binarySearch(double[] a, double key)
        static int 	binarySearch(float[] a, float key)
        static int 	binarySearch(int[] a, int key)
        static int 	binarySearch(long[] a, long key)
        static int 	binarySearch(Object[] a, Object key)
        static int 	binarySearch(short[] a, short key)

This, sort of works. It's a lot of code duplication, but it's all in the library.

Except that a relatively common thread goes like this:

You start by having a generic array that gets passed to said binary search. It works, but it's too memory-hungry when it gets called with a primitive. So, then what do you do?

Well, you go "huh, I could specialize". And then you start specializing, and realize that every function that calls binarySearch with a generic type _also_ needs 8 implementations (would be 9, but no sane person would binary search a boolean[] ).

Basically, its complexity that you cannot even punt off to an external library. If you have code that needs to be able to be called with generics without inefficiencies for primitive types, you'll end up with ~9x code duplication at a minimum.

</comment></comment></comment></comment></comment><comment><USER11> I don't see why people put so much work into GC before even bothering to do any work to avoid objects becoming garbage in the first place beyond simple escape analysis.

</comment><comment><USER14> presentation video: [https://www.youtube.com/watch?v=PnPzeavMjmE](https://www.youtube.com/watch?v=PnPzeavMjmE)</comment>


Ask HN: What are the benefits of a native desktop app vs. a web one? - <AUTHOR>

Nowadays one can do pretty much everything with HTML5 capabilities. Given I've always loved native desktop apps over the Web ones, but can't really list the benefits anymore, can you share your thoughts regarding the topic? What do you prefer? Why? What are the use cases where a native desktop app suits better? Please, exclude video games from the reasoning. Thank you. <comment><USER2> \- Native apps are more or less future-proof. A web app usually needs to be rewritten as browsers and the internet change and evolve.

\- They can be made ridiculously fast. With web you're at the mercy of third- party scripts and browsers.

\- Native is much more energy efficient compared to web apps. You can do a whole lot more with much less power and resources.

\- Native apps are incredibly secure compared to web apps. Unless holes and backdoors were purposefully designed in it, a native app will just run on your desktop. A web app goes through a browser, maybe some extensions, your network, an ISP and finally some third-party servers.

\- Native code is easier to maintain because there are fewer things that can go wrong as time goes by. And it mostly boils down to bugs in the code. With web apps you have to debug web servers, third-party scripts, browsers, resolutions, etc.

\- Native apps don't need a constantly active internet connection.

Web apps do look great. And accessing them from anywhere is a huge plus. But look at the mobile app stores. Look at all the scandals regarding privacy and web servers. Native is preferred because it's faster, more efficient and more contained.

The bleeding edge is never pretty. But it usually blows everything else out of the water.

</comment><comment><USER3> One of the most important cases for desktop applications is when the software must interface with local hardware. This is especially so when that hardware requires large amounts of data throughput, short latencies, and or enhanced security. For reasons including safety, efficiency, and simplicity, not all hardware devices are equipped to communicate over the Internet. Furthermore, when interfacing with local hardware such as robotics interfaces, why in the world would I want the software to run in the cloud rather than locally? Sure, there are a couple of special cases for this, such as when the fruits of the system are to be publicly or remotely accessible (a public webcam, for example), but there are plenty of cases where local access is much more appropriate.

A particularly noteworthy example is automated manufacturing. In this case, it would be nearly absurd to run the control software in the cloud rather than locally. Would you really want an Internet hiccup to cost you a whole batch of units? What about the software that controls a self-driving car? Would you want to ride in that car when the software is running in the cloud rather than locally?

</comment><comment><AUTHOR> I'm sorry for the broadness of my question. When writing it down I was thinking about utility softwares for sharing files or for taking notes that nowadays have usually both client-side applications and web-based ones. Of course specific control software for manufacturing or computation-heavy programs where not in my mind. Regarding File System access, this is one of the biggest advantages I see in a client-side application. Though, we are going towards the sandboxed apps way, and we may lose control over FS even on the client side. HTML may also eventually get more access to FS in future iterations. I do agree also that HTML+CSS+JS is somewhat painful compared to the desktop UI toolkits.

</comment><comment><USER4> I am employed to develop and maintain a desktop application for simulation and modelling with the engineering discipline.

The application has been in development for over 16 years. At inception a web application was totally out of the question.

The application is CPU and RAM intensive. Until recently, this ruled out a web application. Amazon and other elastic compute providers have swung this pendulum a little.

Model size generally exceeds the limits of Local Storage.

The simulation can generate from hundreds of MB to multiple GB of data. While this can now be retrieved over the internet, the time to do so can become an issue. This will be compounded when the user wants to run sensitivity series (ie multiple models with slight variations).

I have built a few CRUD web apps in my time. HTML+CSS+JS was incredibly painful compared to the desktop UI toolkits I have used.

The models built in my application almost always contain client confidential information. Explaining to them that we/they have to send their data to "the cloud" in order to run the simulation is not going to be acceptable.

Whenever I dip my toe into web development to see if things have improved, I still struggle to understand the convolutions required. Compiling CSS? How many JS libraries? I have to sacrifice how many chickens for my app to work right in IE+Firefox+Chrome+Safari+mobile?

As far as I can tell, all the web app stuff is just to sidestep the need to install and update an application? While this might be an important consideration for a (IMHO lazy) IT department or for a communication based tool (Twitter/Facebook etc), it really doesn't make much sense to me for office applications and many other common tools.

Why should I need an internet connection to do desktop publishing? Why do I need to transfer my possibly confidential data to someone else's computer just to transform it? These are things I have always been able to do with a desktop app, without having to compromise on privacy or performance.

</comment><comment><USER5> It really depends on the problem you're trying to solve. I write a lot of programs that are used to troubleshoot other software. I need to be able to access the local file system and the OS.</comment>


Dnsthru.com - web based dns lookups - <AUTHOR> http://dnsthru.com What do you guys think? For now this service provides quite simple (basic) dns lookups for a domains and reverse dns for ip addresses. Any feedback is more than welcome.<p>http://dnsthru.com <comment><AUTHOR> Will fix the konqueror issue.

What is the point? Not everyone knows how to use nslookup, and there are websites out there that charge for this type of service. It is also just a hobby project, brushing up python skills :-).

</comment><comment><AUTHOR> What do you guys think? For now this service provides quite simple (basic) dns lookups for a domains and reverse dns for ip addresses. Any feedback is more than welcome.

<comment><USER2> Doesn't work in Konqueror -- the query line is cleared, but the response never appears.

More importantly: What's the point? I can do this far more easily using nslookup.</comment></comment>


Ask HN: Why did the OH HAI SEXISM thread get killed? - <AUTHOR>

It had ~90 points and ~90 comments.  The issue of how women are treated in tech seems pretty relevant to the HN community.  Why did it get killed? <comment><USER2> Probably because the number of flags it had was significantly greater than 90.

Also - the issue of how women are treated in tech comes up every other day. It is discussed and argued over in each thread, nothing changes, and it comes up again two days later with same results. At some point, there's nothing new to learn about it.

</comment><comment><USER3> probably flagged</comment>


How Austin Beat Uber - <AUTHOR> http://www.nytimes.com/2016/05/12/opinion/how-austin-beat-uber.html <comment><USER2> I don't understand. I've used Uber and Lyft many times throughout the US and they have great service and lower prices. I live in NYC and it is so wonderful having these services is great.

What is this guy talking about?

<comment><USER3> Uber and Lyft chose to leave the Austin market because they object to fingerprinting drivers.

<comment><USER2> I understand that part, but what I don't understand is that the author of the article seemed happy that Uber and Lyft were no longer available in Austin.</comment></comment></comment>


Travis Response to Meltdown Attack - <AUTHOR> https://blog.travis-ci.com/2018-01-08-travis-response-meltdown-spectre <comment><USER2> Even disregarding CPU bugs, Travis CI is a security disaster. Keep your secrets far away from it.

* They use 32-bit id to retrieve an OpenPGP key (meaning that anybody can get their key added to the APT keyring):

[https://github.com/travis-ci/travis- build/pull/1269](https://github.com/travis-ci/travis-build/pull/1269)

* Here they are identifying OpenPGP keys with HTTP URLs (sometimes with 32-bit id, too):

[https://github.com/travis-ci/apt-source- whitelist/blob/maste...](https://github.com/travis-ci/apt-source- whitelist/blob/master/ubuntu.json)

* But none of the above matters much, because they use APT with the --force-yes, which among other things, disables package authentication:

[https://github.com/travis-ci/travis- build/commit/c4d15425f7b...](https://github.com/travis-ci/travis- build/commit/c4d15425f7b1)

[I reported the --force-yes bug to the security team in October 2017, but AFAICS nothing changed nice then.]

* [https://github.com/travis-ci/apt-package-whitelist](https://github.com/travis-ci/apt-package-whitelist) is just bizarre. They try to to detect " _malicious_ or goofy bits" with grep. What.

And these are just things I stumbled upon randomly...

If you're using Travis CI only for testing stuff, you should be fine. If you're using it for deployment, you're doing it wrong.</comment>