Signal partners with Microsoft to bring end-to-end encryption to Skype - <AUTHOR> https://signal.org/blog/skype-partnership/ <comment><USER2> The thing is. How will Microsoft and Skype handle backdoors now. As far as I understood the reason Microsoft broke Skype so badly was because they used centralised servers with backdoors for countries who wanted them. Not always the good countries.

But this. This baffles me. Deeply.

<comment><USER3> Microsoft used centralised servers because the Skype prior to that was a curse to mobile devices running on battery power. Particularly cellphones.

Skype worked as a p2p network, where some peers where marked as super peers and would help with peers behind firewalls (UDP-holepunching), and routing through the super peer. If your phone became a super peer, you could expect to essentially work like a server, with the "benefits" of increased bandwidth usage and power usage. Not exactly what you want as a mobile user.

So Microsoft had to change the architecture (which wasn't designed with mobile devices in mind) into a more centralised approach that could work with mobile devices.

<comment><USER4> This doesn't explain why after the change Skype started routing calls between machines on the same LAN through Microsoft servers.

<comment><USER5> It is difficult to make a p2p app on a mobile device work _even between devices on the same LAN_.

A simplified explanation: Mobile devices will often ignore almost all incoming network traffic to reduce battery usage. The only way to reliably communicate with the device is through a centralized push notification service (e.g. APN and GCM).

<comment><USER6> I tried to make a demo app to perform very simple HTTP p2p and found this difficult on iOS. When an app is backgrounded you have ~17 seconds to stop execution or the OS will kill any active threads UNLESS the activity is among the Apple permitted exceptions. (Note that there is no iOS SyncThing app). In Android it is possible to use IntentService as a single threaded background HTTP server. It works for the most part but can be a little flakey. I'm not 100% sure how the OpenGarden SDK accomplishes this but I'm interested. My guess is that it makes heavy use of Bluetooth, which is one of the Apple permitted exceptions to the backgrounding rule.

<comment><USER7> > there is no iOS SyncThing app

There's fsync() but it only syncs when you launch the app - what I think you meant is "there's no continuously syncing iOS SyncThing app".

<comment><USER6> Yes, thanks for the clarification. Continuous/background syncing is the difficult task.

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> Just to get a sense of the track record here, Signal Protocol powers:

* Facebook/WhatsApp

* Google/Allo

* Microsoft/Skype

* Signal

Signal is also the basis for the protocols for Wire (Proteus) and Matrix (Olm).

<comment><USER9> Can't say I am really happy about it. The more the Signal protocol is used the more money is spend on attacking it.

<comment><USER10> "If you haven't been hacked by the Chinese you got to ask yourself, does the shit you're doing really even matter?"

I prefer for Signal to be attacked and to matter than to be unattacked and to be irrelevant.

<comment><USER11> Indeed, it's incredibly selfish and naive to depend on security by obscurity here when you can significantly up the bar from basically one step above full- text all-access for every global spy agency... to a serious encryption system.

Regardless, the exposure ship sailed long ago anyway when WhatsApp with their billions of users, including plenty of terrorist groups in the middle east and africa, decided to adopt the platform.

Additionally, breaking crypto systems like this in practice, if possible, pretty much _always_ means targeted attacks. Which is a significant difference from the type of mass surveillance which Skype et al currently allow.

Even Blackberry bragged about building a system for the NSA/CSEC to provide real-time data access to every BBM being sent in the Toronto area during G20. It's very likely that Skype has a similar system and this change (should) make that fundamentally infeasible.

</comment></comment></comment></comment><comment><USER12> Maybe a good opportunity to remember what is the main mission of Signal/Open Whisper Systems: they realized at the time that it was extremely complicated for non-technical people to use tools secure enough and respecting your privacy (PGP...), and that the tools used by everyone (Whatsapp, Messenger...) were popular because they were fun and easy to use.

They believed that everyone should be able to have an easy way to communicate with everyone, with a nice UI/UX, AND have an app secure and respectful of your privacy. And this, leaded to the Signal protocal and the Signal app .

<comment><USER13> While on the topic of things to remember, also remember that they're making compromises in order to keep things hidden from users.

The Signal protocol may be sound, but as we've seen with today's WhatsApp news, there are still implementation-specific compromises being made. Not to mention that many of those companies ship a closed source product. They could publish a spec of what should be going over the wire to make it auditable without needing to go all open source, but they don't do that either. Things are really kept closed. I am not sure whether it's a net positive or a net negative when another user joins WhatsApp or a similar service, versus Telegram where encryption is opt-in but at least it's open source and not leaking metadata to BigCorps whose profit model is knowing you.

<comment><USER12> Though I understand your points, Signal is a protocol, you can implement and build things around it the way you want. Sure you can say that WhisperSystems should enforce a secured architecture around it, but it is not their job, and I doubt Whatsapp, Messenger and Skype would be using Signal if they were forcing that.

And I would definitely not say Telegram is the best tool here [1][2]. If you want something really secure, I would recommend the Signal app [3].

In the end, it is always who you decide to put your trust in.

[1] [https://security.stackexchange.com/questions/49782/is- telegr...](https://security.stackexchange.com/questions/49782/is-telegram- secure#49802) [2] [https://eprint.iacr.org/2015/1177.pdf](https://eprint.iacr.org/2015/1177.pdf) [3] [https://signal.org/](https://signal.org/)

</comment></comment></comment><comment><USER14> Wikipedia: "Signal relies on centralized servers that are maintained by Open Whisper Systems."

Would it be possible to use the same open source proto and crypto that Signal chose, but in a way that does not rely on _third parties_ to run servers, such as OWS, WhatsApp, Facebook, Google, Microsoft, etc.?

<comment><USER15> Yes, but decentralization is a difficult problem. I think eventually it will become like email, where everyone just uses GMail.

NAT, Firewall, etc all make it very difficult to do true P2P as well.

<comment><USER16> >I think eventually it will become like email, where everyone just uses GMail.

Or Outlook(Office365), Ymail, Zoho, GMX, iCloud, Yandex, Proton.

<comment><USER17> Or Posteo, Mailbox.org, Tutanota, Riseup.net, etc.

<comment><USER18> Or self-hosted.

<comment><USER19> A lot of people are listing plenty of alternatives for email, but the majority of people aren't going to be this broad in email usage.

</comment></comment></comment></comment></comment></comment><comment><USER20> Over the last couple of years, there's been huge progress in making end-to-end encryption practical for "real world" apps -- especially stuff running in web browsers. Signal is part of that, of course, as is WebRTC, a standard that to its great credit did an excellent job with the encryption pieces from the very beginning.

We built our video calling app, Daily.co, on top of WebRTC and so we could relatively easily include end-to-end encryption from the day we launched.

A few things make e2e harder as you scale up. One is that metadata gets more and more important, for a lot of reasons, as you add features and try to grow a business. As has been pointed out here, Skype _was_ end-to-end encrypted in the pre-Microsoft days.

Another challenge is that it's a lot easier to build e2e encryption for mesh network topologies than if you're routing things through central servers. For video calling, mesh networking breaks down as you add more users to a call. For us, that means we're e2e encrypted up to 4 people in a call. After that, we're forwarding video streams through our servers in the cloud, and we don't do e2e encryption anymore. (Not that we wouldn't like to, it's just a much heavier development lift.)

<comment><USER21> Why not just forward the e2e encrypted video streams through your servers? Encrypted or not, wouldn't the amount of traffic be the same?

</comment></comment><comment><USER22> If anyone from Skype product-team is reading this - Please bring back the ability to dial from the local contacts, without having to Sync my entire address book. This is my minimum request.

If possible, just put the old Skype interface back. The new design with all its jazz-matazz and sparkles have made the product unusable. It is not a first-world unusable I am talking about. The product is not usable, as in I am moving away from Skype after being a customer who has given thousands of dollars in business over the years.

How did the new Skype design pass user feedback reviews? Honestly asking to learn.

<comment><USER23> it's unbelievable how badly skype functions as a PC IM client. it just keeps getting worse and worse. there's zero reason an IM client should be using hundreds of megs of ram and a hideous, garish UI that you can't turn off.

the only workaround i found was the pidgin skype-web plugin, which can be a little buggy, but it's far preferable if all you use skype for is IM.

<comment><USER24> Not sure if it's the same client, but the Skye for Business client is also just as bad.

I used to work on a virtualised Windows machine (work mandated) and the only usable accessibility option was a high DPI setting. The old Lync client worked perfectly with this, obviously dating back to an age where they cared about accessibility. Skype for business completely ignored system DPI settings, made certain things huge with other things being tiny.

The spacing was the biggest issue - literally 30-40% of my 43" monitor would often be taken up by spacing around icons etc on the app.

It really felt as though they just stopped putting as much effort into UX, accessibility, usability. Lync felt like a professional IM client, Skype for Business felt like a half baked bodge.

</comment></comment></comment><comment><USER25> Now if only they can partner with someone to get chat working as well as it used to or get syncing between multiple devices working...

<comment><USER26> This. I wish Signal's own native apps had the user experience of Telegram.

<comment><USER27> or that telegram just used the signal protocol; either is fine in my book

<comment><USER26> Yes, that would be great as well but I've long since given up on that happening.

</comment></comment></comment></comment><comment><USER28> I made a request for e2e to Discord a while back. It became one of the most voted suggestions but got denied. Maybe this will push them to take another look.

<comment><USER29> I'm not knowledgeable about this -- is signal's e2e encryption even viable for a service with N users, and one where new users are being added?

Put another way, if I have a group with 10 users and it's encrypted and then another person joins and can see the old messages, was it actually securely encrypted in the first place?

<comment><USER30> > if I have a group with 10 users and it's encrypted and then another person > joins and can see the old messages, was it actually securely encrypted in > the first place?

Sure, why not? Assume, just because it's the first thing that came to mind, that message history is encrypted using GPG and every message is encrypted with everyone's individual public keys. When a new user joins, one user just encrypts the message history with the new user's public key.

Or am I missing something? Do the existing users in the group _not_ have access to their own message history?

<comment><USER31> You're not missing anything, this is definitely possible to do.

That said, encrypting every message with everyone's public key doesn't scale, so instead you'd likely want to generate a symmetric key for the group and as new users are invited the inviter would encrypt the group key with the invitee's public key. That makes it trivial for new users to view history, and as new chats are posted they only need to be encrypted with one key.

</comment></comment></comment></comment><comment><USER32> skype had this already before ms. ms opened skype up to government ages ago so this doesn't mean much of anything, there are backdoors. [https://youtu.be/J1q4Ir2J8P8?t=2880](https://youtu.be/J1q4Ir2J8P8?t=2880)

<comment><USER33> Yeah, I was gonna say Skype communication protocol was proprietary and encrypted until Microsoft bought them and rewrite a lot of it.

</comment></comment><comment><USER34> Skype is proprietary software. Even if it encrypts my messages E2E, it still needs to display them with its proprietary UI and see them in plain text. There's no way I could trust that or recommend anyone else to.

The thing about E2E encryption is that you still need to have reasonable trust in each E.

<comment> Digital-Citizen I concur.

It's interesting that so many people who claim they understand what the software is doing don't appear to understand that proprietary software is always untrustworthy. Users have no real control over proprietary software, no matter how technical and willing they are to change their copy of the software.

Therefore it doesn't matter which apps are installed on a proprietary OS. The proprietary OS (or possibly some hardware beneath it) is untrustworthy. Every keystroke, drag/gesture, location change, camera/mic input, and more pass through proprietary software before they get to the ostensibly trustworthy app.

</comment></comment><comment><USER14> More than a few comments trying to rationalize why Skype is different under Microsoft. (Supernodes are controlled by Microsoft, not users.) Also some comments exhibiting misconceptions about the relative feasibility of peer-to- peer networking today compared to the past.

Some users want peer-to-peer networks that are controlled by users, not third parties such as Microsoft, Facebook, etc. It seems that corporations are also interested in such networks. But they just want the users, not the ability to exclude third parties (they _are_ a third party).

What is important for these interested users is that _Skype changed_ and _how_ it changed, not _why_ changed.

No explanation, rationale or excuse is a substitute for a peer-to-peer network _that is controlled by its users_ , not third parties.

We know why these user-controlled networks get acquired (or copied) by companies: because they work and they attract many users.

</comment><comment><USER35> You know, before Microsoft Skype had end to end encryption.

<comment><USER36> For a minute, I thought your comment had simply trailed off without finishing your thought.

Then I realised it wasn't "Microsoft Skype", but rather, "before Microsoft, Skype had..."

<comment><USER35> I see, thanks for the info. Comma setting in English is rather unexpected for me. It works different in German, but I'll try to adapt. Maybe in English you would also say "end-to-end encryption" rather than "end to end encryption", is that correct?

<comment><USER36> Both are commonly used, although "end-to-end" is correct as it indicates a compound adjective - a phrase acting as a single adjective affecting the noun ("encryption") that follows it.

</comment></comment></comment></comment><comment><USER37> While they are it, could they make the Skype application an actually reliable one? I find WhatsApp video and call quality far superiour to that of Skype, which has years on WhatsApp. Not only is the video and call quality subpar, the app itself often behaves in odd ways. Personally, I've come to dislike Skype greatly.

</comment><comment><USER38> Great news. More funding for Open Whisper to provide us Signal.

<comment><USER39> They could get 100x the amount they've gotten so far from these companies if they had launched their own crypto-asset, which is probably by far the best way for an open source project to get funding.

<comment><USER40> Moxie Marlinspike is involved in: [https://www.mobilecoin.com/](https://www.mobilecoin.com/)

</comment></comment></comment><comment><USER41> How does Signal's chat security compare to Keybase.io's chat? Keybase has more features, but if it's not as secure, then it doesn't matter.

<comment><USER42> Keybase doesn't support forward secrecy. The signal protocol does, though I can't say whether that's true for Skype's implementation of the protocol

<comment><USER43> Some docs here: [https://keybase.io/docs/crypto/chat](https://keybase.io/docs/crypto/chat)

To add a little bit, avoiding forward secrecy was a design decision. We wanted to support adding and removing devices from your account (including removing all of your original devices, if you want), and we wanted new devices to be able to read your message history. I think those two things put together are in conflict with forward secrecy.

That said, we'd like to allow you to turn off history for some messages, and it would be nice if you got forward secrecy for those messages when you did that. We're currently in the middle of figuring out how that's going to work. One of the open problems is this sort of situation: If I have 5 devices, and one of them is a laptop that's been in the closet for 3 years and won't ever rotate its keys again, how do we avoid making that laptop a giant hole in my forward secrecy guarantees?

<comment><USER44> It seems like a reasonable design decision to have a time horizon beyond which a device which has been out of communication loses access to more recent messages. That could be significantly less than 3 years - probably a week or two is fine.

</comment></comment></comment></comment><comment><USER45> Does this mean Skype will stop cooperating with governments demanding chat history disclosure?

<comment><USER46> They'll still cooperate but they'll have much less data to provide.

</comment></comment><comment><AUTHOR> I can't help but read this with a cynical voice in my head: Skype used to have an incredibly bleeding edge P2P E2E encrypted protocol, and Microsoft threw it all away. Some might say for good reasons (mobile use case, supernodes straining routers, legal wiretap compliance issues); I'd respectfully disagree and observe that there's just going full-circle.

<comment><USER47> Yeah, Microsoft moving to a model where "lawful intercept" is no longer possible would be hugely surprising.

<comment><USER48> They can still lawful intercept. All they have to do is push a MITMed version to suspected law breakers. It's going to be closed source just like WhatsApp, so of course, it will be easy for LE to defeat while shedding crocodile tears publicly about how it's uncrackable.

<comment><USER49> How could Skype target binaries to individual users?

<comment><USER50> They don't need it. They can simply turn a switch and disable E2E between parties under investigation.

Remember this is closed source and you're using a central infrastructure.

</comment></comment></comment></comment></comment><comment><USER51> My skype broke (empty everything) on ubuntu,android,web and only works on windows old installation with upgrade.

<comment><USER52> Same here. Until now, I assumed I'd somehow screwed up my registration on Android.

</comment></comment><comment><USER53> It is probably opt-in and a crippled version of the default chat interface.

This way the few users of the feature will be visible like a Christmas tree in a dark forest and the general userbase will think it's their laziness not to use encrypted Skype.

That's also how Facebook implemented it in messenger.

</comment><comment><USER54> Skype complies with at least court-ordered wiretap requests (they were forced to when MS bought them). How exactly will that work without "e2e but with key escrow"? And "e2e with key escrow" is basically the same as "plaintext"

</comment><comment><USER55> I abandoned Skype after Microsoft bought them out, I don't miss them even a little. If Microsoft could buy Linux they would probably break it as well, thank goodness its not for sale

</comment><comment><USER56> Does this mean I'll need to give Skype a verified phone number and some version of all my contacts for … reasons. Reasons that are, trust us, so important that we can't let you opt out of them, but that have nothing to do with us keeping your phone number.

I like everything else about Signal, but won't use it as long as they mandate that you tie your account to a verified, non-throwaway phone number. That's asking for more trust than I'm willing to give.

<comment><USER57> This is Signal the E2E protocol (also used by WhatsApp, Facebook Messenger and Google Allo), not Signal the app. The number requirement is part of the app. I assume your identifier in Skype will be your Skype name/MS Account.

</comment></comment><comment><USER58> Does mean the consumer Skype or the corporate Skype?

<comment><USER59> Article refers to consumer Skype, not SkypeForBusiness.

</comment></comment><comment><USER60> Will Chinese citizens be receiving this update?

</comment><comment><USER61> What's the point ? There's still backdoors in Skype anyway, lol.

<comment><USER62> Indeed. The latter end of end-to-end may not be the end intended by the user.

</comment></comment><comment><USER63> I would much more prefer if they would fix such basic things as missing front/rear camera switch in video call or inability to send more than one photo at same time.

</comment><comment><USER64> Would be cool if MS would just drop Skype and adopt Signal.

</comment><comment><USER65> This means we can still trust Signal right? That's the app I use because some large security experts said it was one of the best.

<comment><USER66> Nothing about Signal itself is changing. This is Microsoft adopting the Signal Protocol for a new feature in Skype.

</comment></comment><comment><USER67> While on topic of Skype: My God the latest Skype design update is abysmal, whoever come up with that horrible mess should be fired and never touch any design or management role at all.

And it's not only design but the ux is horrible too, when you switch between conversation - it does not focus on the chat box field so you can start typing right away, instead you have to click it first. This is a basic stuff for a chatting app and whoever missed that is apparently clueless about ux

Edit: also if any of Skype iOS app developers is reading this - on iPhone X when you accidentally click top left corner of the phone where the clock is - for some inexplicable reason the whole conversation scrolls all the way to the top, which is incredibly annoying!

<comment><USER68> > on iPhone X when you accidentally click top left corner of the phone where > the clock is - for some inexplicable reason the whole conversation scrolls > all the way to the top, which is incredibly annoying!

That's by design and standard iOS behaviour (it has been there for years). Tapping anywhere on the system toolbar (i.e. where the clock, wifi, signal strength indicators are) brings the main scrollable view to its "topmost" position.

<comment><USER69> I just tried it on this webpage, and you're correct!

I really dislike behavior like this that is useful to a few people but is far more likely to be encountered accidentally and feel buggy.

<comment><USER70> You're kidding right? It's been a feature of iOS for what feels like forever, and it's literally the one thing I miss most when switching to Android. I have witnessed first hand, very un-tech savvy iOS users using the feature, as well as people confused (my daughters) why the same action on Android doesn't product the same result.

I thought this was one of Apple's "protected" features that everyone used/loved - guess not!

<comment><USER67> I agree it's useful on web pages but there has got to be an option to disable that in app conversations.

</comment></comment></comment></comment></comment><comment><USER71> Skype is part of X-KEYSCORE, so NOPE>

</comment><comment><USER72> Screw Skype. That is all

</comment><comment><USER73> Ugh, people are still using Skype?

<comment><USER74> Still looking for something that is free, does (persistent) group chat, voice calls, file sharing, and group voice calls. Bonus if it shares screen.

Have tried various alternatives but actually not yet found anything that works as good as skype, despite having terrible UX. The sound quality is second to none and file sharing etc usually "just works".

<comment><USER75> Discord does all of this, though it's not E2E encrypted, and does it all with a fantastic UX.

<comment><USER74> For some reason having the (non business) Skype is not hard to sell in a big "old school" enterprise, because it's microsoft.

I'd have a much harder time selling Discord because of how they market themselves. They could make "discord for business" that is _literally the same app_ and have great success.

Other than that my only complaint is that it's a bit electron-y.

<comment><USER76> > They could make "discord for business" that is literally the same app and > have great success.

Discord is sometimes known as "slack for gamers" so you're effectively describing Slack :)

<comment><USER74> Yes. But slack, like HipChat thinks I want to pay 500MB RAM and $7 per user per month to chat and voice call.

Effectively all I want is a lean chat/voice app that's also ideally free. This is pretty easy - but when you throw in the other Skype features (group voice, screen sharing, mobile apps) as requirements, it's actually pretty hard to match in the free tier.

Skype is $0 regardless of company size and the resource use is pretty reasonable.

On that topic: Why aren't more companies using discord instead of Slack? Looking at feature matrices it seems discord does in the free version a lot of what Slack does in paid (e.g group voice)?

<comment><USER77> One big reason not to is due to corporate policies, mostly could be summed into legal matters, such as privacy, usage rights etc.

</comment></comment></comment></comment></comment></comment></comment><comment><USER39> This is still an _optional feature_ , so like 98%+ of the users will not use, and the other 2% probably won't use it all the time either.

Right now only Signal and WhatsApp enable the end-to-end encryption by default.

Microsoft should remember that they can't use the excuse that "they can't provide law enforcement with the messages" unless the users were also using the E2E mode/Private Conversations.

It would also help if Microsoft stopped supporting voice calls in its main client, too, as that automatically qualifies it as a "telecom provider" in many countries, which means it automatically falls under the same lawful intercept laws that affect telecom providers.

Unless these changes are made, then this news hardly changes anything in regards to Skype messages being intercepted.

</comment><comment><USER78> yeah... um. Why would Signal (OpenWhisper) sully their name by coordinating efforts with MS? MS doesn't have a great track record at being an advocate for privacy or user security...

<comment><USER79> The mission of OpenWhisper is to bring e2e encryption to the masses, not to make Signal the messenger the dominant IM tool. They succeed spectacularly by making the popular IMs adopt Signal the protocol.</comment></comment>


On Max Perkins, One of America's Greatest Editors - <AUTHOR> http://lithub.com/on-max-perkins-one-of-americas-greatest-editors/ <comment><USER2> I read Berg's biography back when I was a noob editor, and I recall it being a good read. It is not an exaggeration to say that several generations of editors were influenced by him (an example that would be almost impossible to emulate today). The 'new' authors mentioned--Paton (Cry the Beloved Country) and Jones (From Here to Eternity)--ended up being the last books he edited (spoilers: he dies at the end).</comment>


Tesla Model Y takes the ultimate off-road challenge and wins - <AUTHOR> https://www.teslarati.com/tesla-model-y-ultimate-off-road-challenge-wins/ <comment><USER2> Nice! But from the title I thought it'd done something like Hells Gate in Moab

[https://www.google.com/search?q=moab+utah+hells+gate&oq=moab...](https://www.google.com/search?q=moab+utah+hells+gate&oq=moab+utah+hell&aqs=chrome.2.0j69i57j0l2j0i22i30.7671j0j4&sourceid=chrome&ie=UTF-8)</comment>


Ask HN: How can I be more effective? - <AUTHOR>

I've spent the last year radically overhauling a lot of things about my life, mostly in the areas of self-discipline, building good habits, and time management. I feel really good about what I've been able to accomplish, but I feel a plateau starting to emerge. I'm not sure I can squeeze much time out of my schedule without it being unsustainable (I'm not willing to sacrifice my health, sleep, or relationships). So I now have a lot more productive time than I did before, but I'm wondering how I can best leverage those improvements for even bigger gains in the future.
How can I be more efficient with the time that I have? (doing things better)
How can I be more effective in general? (doing better things)
I realize these questions are very general, but I'm interested in hearing a lot of different perspectives and whatever general principles you've noticed from years as entrepreneurs, developers, and humans.
PS - This might sound like I'm sacrificing life for work, but I'm really not. I'm married and spend lots of time with my wife, friends, and family. I'm working on my pilot's license. I exercise, try to eat well, get enough sleep, and take time to read great literature, etc. <comment><USER2> Well ... it sounds like you're plateaued in the "work harder" area (although incremental improvements can no doubt be found). Time to focus on "work smarter"?

Explore powerful tools, from math per se to the more mathematically sound languages and systems?

</comment><comment><USER3> best advice I ever had on time mgmt was to read 'how to live on 24 hours a day ' www.gutenberg.org/etext/2274 - excellent advice even though written over 100 years ago.

only other thing would be to see how a VA would work to make your life better

but like all things in life you will reach a plateau, you can only refine something so far!

good luck, sounds like you are doing well!

<comment><AUTHOR> Already have a VA :)

Thanks for the book recommendation!

<comment><USER3> its a great book, but the jist of it is wake early, spend time doing what you really want to, learn when you can.

trying to figure myself at the moment exactly what a VA could do for me. can't open my work email to them until I am self employed so just trying to figure whatever they could streamline for me.

my other time saver tip is instapaper and instantly for android. read the blogs I love when I have the time.

<comment><USER4> what does VA stand for?

<comment><AUTHOR> virtual assistant

</comment></comment></comment></comment></comment><comment><USER5> Work on your health. Having physical energy and mental clarity is rooted in physical well being and both of them add to productivity.

Read books like "The 7 habits of highly effective people".

Good luck.</comment>


Rationality for Entrepreneurs: Awesome workshop in the Bay Area - <AUTHOR> http://appliedrationality.org/schedule/ <comment><USER2> I went to their earlier workshop in July. I think it was probably the most useful week of my life in terms of exposure to things I could be doing to be more productive.

Since attending, I've been getting considerably more work done (my brother has noticed too). Anna Salamon's class on fungability prompted me to change the direction of one of our businesses. Plus, the CFAR people are just fun to be around!

<comment><USER3> Anna Salamon wrote an article on LessWrong, "Humans are not automatically strategic," which changed my life:

[http://lesswrong.com/lw/2p5/humans_are_not_automatically_str...](http://lesswrong.com/lw/2p5/humans_are_not_automatically_strategic/)

I wrote a bit about it 2 weeks ago here on HN:

<http://news.ycombinator.com/item?id=4702484>

</comment></comment><comment><USER4> I attended their May workshop. It was exceptional and it seems like they have improved on the format since. If you are the sort of person who is interested in such things it is probably worth your time and money. (Or you can just value it as a highly interesting and fun vacation.)

If you're in the Bay Area you might want to beta test the quality of the instruction for free at their Berkeley headquarters; they often have workshops in the evenings there. The next is on Thursday. If you sign up for the newsletter on their site you'll get notified.

</comment><comment><USER5> I went to one of their minicamps and learned some amazing mental tools like fungibility, environment design, and Bayes law.

</comment><comment><USER6> I know Geoff Anders, who is teaching a few of the sessions, and he is awesome. I live in NY and don't have time to fly out for this, but would be very interested if I lived out there.

<comment><USER7> I second Geoff Anders being awesome. leverageresearch.org doesn't do Leverage Research (his org) justice, but there's plenty of interest there anyway.

</comment></comment><comment><USER7> I went to their July "minicamp", which was an awesome experience.

</comment><comment><USER8> I'll be there.

</comment><comment><USER9> Sadly, I don't have $3,900 to shell out. It would be wonderful if there was a similar workshop that was more openly accessible.

<comment><AUTHOR> Ask about a scholarship

<comment><USER9> Do you know how the scholarship is income-dependent? Because, obviously, my parents could afford it (I'm on a gap year(s) out of high school), but I personally could not (working at a startup).

<comment><USER10> Hi Brian, I'm the president of CFAR; we still have some scholarship money left so I'd encourage you to fill out the (short) application form today. If you're a great fit for the workshop, we can figure out a price that works.

</comment></comment></comment></comment><comment><USER11> This reminds me of a quip in a New York Times article about Amazon's exclusive book, Timothy Ferriss's "The 4-Hour Chef:"

 _Those books about finding success without trying too hard were a particular hit with young men, who identified with their quasi-scientific entrepreneurial spirit._

The schedule reads like a table of contents from a self-help book. That could be a good thing.

I'm just surprised how much traction lifestyle instruction gets with an unlikely demographic: 18-24 year old men with access to unlimited concrete learning materials online.

</comment><comment><USER12> Is there a way to read up on the things taught in the workshop in particular on the more applied/practical material?

</comment><comment><USER13> Please correct the title - Entrepeurs is a pretty unflattering misspelling

</comment><comment><USER14> Is there going to be an east coast version in the future?</comment>


Mysterious explosions keep happening in Iran - <AUTHOR> https://www.vox.com/2020/7/17/21325985/iran-israel-explosion-natanz-nuclear-missile <comment><AUTHOR> The Natanz nuclear facility, which had an explosion on July 2nd, is the same site which was struck by Stuxnet in 2010.</comment>


How Discord Stores Billions of Messages Using Cassandra - <AUTHOR> https://blog.discordapp.com/how-discord-stores-billions-of-messages-7fa6ec7ee4c7#.dzqq7q4o7 <comment><USER2> These kinds of write-ups offer valuable insight into a popular project's requirements and decision-making, and are some of the most instructive resources one can find: these show not only the kinds of challenges one has to face at scale, but also how architectural choices are made.

It's far more valuable to understand _why_ Discord uses Cassandra than to merely be aware they do.

Out of curiosity, did you consider HBase and Riak? Did you entertain going fully hosted with Bigtable? If so, what criteria resulted in Cassandra winning out?

<comment><USER3> Let me take a stab at that.

Riak is not a good model since its more a blob store and we wanted to simply range scan through messages rather than sharding blobs (Cassandra is REALLY good at this).

HBase would have been fine for this model, but the open source version of HBase has much lower adoption than Cassandra so that was a big factor. We also don't care about consistency and HBase is a CP database, we prefer AP for this use case. As far as using GCP's BigTable (HBase compat), we made this decision before we moved to GCP, but we are also not fans of using platform lock-in. While BigTable has the same API as HBase we would hate to go to an less widely adopted version where we have a hard time getting community support if we decided to leave GCP.

Hope that helps.

<comment><USER4> > As far as using GCP's BigTable (HBase compat), we made this decision before > we moved to GCP, but we are also not fans of using platform lock-in.

Did you consider GCP Datastore as well?

It has strong consistency for a single "entity group", but eventual consistency for queries on multiple entity groups.

So by storing data only relevant to a single user in an entity group, you can have strongly consistent, atomic transactions on that group (albeit limited to 1 tx/s), and at the same time do global queries on all user data with eventual consistency.

<comment><USER3> The pricing model does not fit our needs, and that is even more locked in than the BigTable variant.

<comment><USER4> I'm happy to hear you dropped it for non-technical reasons, since I'm asking because I've chosen Datastore for an app because I care less about vendor lock-in than ease of operation, and it fits my pricing model perfectly, due to the app in question receiving (Bitcoin) payments that are charged a fee on a per-request/payment basis.

 _Hint: if you have technical reasons for avoiding GCP Datastore I 'd be very interested in hearing about them_

<comment><USER5> Google Cloud is the least geo-distributed provider around. Which is a major problem if your use case has requirements around (a) latency and (b) data locality due to legal requirements.

In 2017 they will finally have datacenters in Sydney, London, Singapore, Frankfurt etc.

<comment><USER6> This is one area where Azure is leading with both Azure SQL and DocumentDB supporting geo-replication.

<comment><USER7> nope since azure is extremly expensive and also you need several accounts for different regions. i.e. you can't create servers in germany with a whole new account / credits / support.

<comment><USER6> This is about capabilities, not price. Azure Germany is the only one that requires a different account due to German legal issues. The rest of the datacenters are all connected from the same account.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> I use Discord a fair amount, and something that annoys me about it is that everyone has their own server.

I realize this is a key part of the product, but the way I tend to use it is split into two modes:

\- I hang out on a primary server with a few friends. We use it when we play games together.

\- I get invited to someone else's server when I join up with them in a game.

The former use case is fine but the latter annoys me. I end up having N extra servers on my Discord client that I'll likely never use again. I get pings from their silly bot channels (seemingly even if I turn notifications off for that server/channel), and I show up in their member lists until I remove myself.

I wish there was a way to accept an invite as "temporary", so that it automatically goes away when I leave or shut down Discord. Maybe keep a history somewhere if I want to go back (and the invite is still valid).

Aside from that, it's a great product and really cleaned up the gamer-focused voice chat landscape. It confuses me that people will still use things like TeamSpeak or (god help you) Ventrilo when you can get a server on Discord for free with far better features.

Now that I posted this, I realize this has little to do with TFA. Sorry.

edit: formatting, apology

<comment><USER3> We have plans to make using temporary sessions in games a much better experience so look out for that in the future.

<comment><USER9> There's also a "Grant temporary membership" option when creating the invite that will automatically kick users when they disconnect unless a role has been assigned to them, but having that as an option when accepted would be cool.

</comment></comment></comment><comment><USER10> Discord seems to me like it has a very polished user experience, and it's no surprise that users are trashing programs like Skype in favor of Discord when it is better in every area.

Discord seems to take security seriously, as they should, but I'm curious about their stance on privacy and openness. For example, I wonder if they would consider:

\- Allowing end-to-end encryption to be used between users for private communications

\- Allowing users to connect to Discord servers using IRC or other clients (or, at least having an API that easily allows this)[1]

\- Allow users to have better control over their own data, such as providing local/downloadable logs so that they can search or otherwise use logs themselves

Discord is definitely succeeding within the gaming market, but I'm curious what other markets they would like to take a stab at.

[1] I'm aware Discord has an API, but if I understand it correctly, normal users cannot easily use Discord from anything other the official Discord apps, as this API is specifically for Discord 'bots'. I see there's a discord-irc bridge, but not much more than that. I may be incorrect on this.

<comment><USER11> > Discord seems to take security seriously

Any app that has voice turned on whenever it detects sound by default, without prompting the user on installation, doesn't take security seriously.

I mean, unless you expect a communications app, running in the background, to share the conversation you're having in your room, without telling you, with everyone in every channel, until you discover it in your user preferences.

(I'm going to assume you're going to misunderstand what the issue here is. It listens by default, like when you install, and you're not prompted that it's the default. Contrary to every other communications or microphone app in existence, save for ones that are designed to spy on people).

<comment><USER12> I don't think this is a "security" issue as much as it's a usability or privacy issue, and I don't think it's an example of Discord being evil.

For a start, it's not quite "on install", but after joining your first voice channel. The issue comes from the interaction of a series of reasonable steps that on the whole result in an unfortunate experience for some people. The problematic series:

* By default, Discord uses voice detection to determine when you're speaking, as opposed to push-to-talk. This feature makes perfect sense.

* By default, Discord configures itself to start up on login. This feature makes sense. (I personally immediately turn that option off, but I don't resent its inclusion.)

* When started, Discord rejoins any voice channel you were in when Discord was last exited. This feature also makes perfect sense. [Edit: Apparently this is no longer true, and Discord will only rejoin the channel if you were active within the past 5 minutes.]

Essentially the result of these design decisions in series is [edit: was] that if you install & use Discord, and fail to manually disconnect from your voice channel, next time you start your computer Discord will automatically join your last channel and broadcast any loud enough audio in the same room as your computer to the voice channel.

There are a few mitigating factors, too: Discord is pretty obviously open and on the screen when this happens, and it does show your active voice channel, and it does show an activity indicator when you're broadcasting.

<comment><USER13> > Essentially the result of these design decisions in series is that if you > install & use Discord, and fail to manually disconnect from your voice > channel, next time you start your computer Discord will automatically join > your last channel and broadcast any loud enough audio in the same room as > your computer to the voice channel.

It's worth noting that Discord no longer does this if you've been away from the voice channel for more than 5 minutes. The feature was intended to autoreconnect you when the app was restarted due to updates and such, not to cause people to accidentally broadcast themselves on system start.

<comment><USER12> Ah, excellent. I wasn't aware they'd added a timer to the channel reconnection. That's an elegant way of solving the problem without compromising the important part of the features.

</comment></comment></comment></comment></comment><comment><USER14> It's really interesting to see that you're using Cassandra for this. IIRC, Cassandra was created by Facebook for their messaging, and realized that eventual consistency was a bad model for chat, so they moved to HBase instead. (source: [http://highscalability.com/blog/2010/11/16/facebooks-new- rea...](http://highscalability.com/blog/2010/11/16/facebooks-new-real-time- messaging-system-hbase-to-store-135.html))

The tombstone issue was really interesting ! Thanks for sharing.

<comment><USER15> You can have strong consistency in Cassandra - ING gave a talk at Cassandra summit 2015 on their multi-DC Strongly consistent use cases

Cassandra let's you choose - per query - how many replicas must ack the query. Strong consistency is just a query parameter away.

<comment><USER16> Need to be careful about the wording, "strong consistency". I dislike that datastax uses that in their documentation, because it's misleading and really confuses people. There is no commit protocol in place - Cassandra is still an AP db under the hood, so even having multiple replicas acknowledge doesn't mean the data is consistent. For that you need paxos or something similar. This becomes very obvious if you are doing updates from multiple sources to the same key.

<comment><USER15> You do realize that Cassandra implements paxos and has a CAS system right?

</comment></comment></comment></comment><comment><USER17> It is fascinating that more and more people are using Cassandra. DataStax believes they have fixed problems with prior guarantees claims that were exposed by Jepsen. But there has been no official Jepsen testing since.

On the topic of looking at Scylla next, I wonder why did the team not just start out with it to begin with. Also, are they people with experience running both. How is the performance? And what is the state of reliability?

<comment><USER3> The problems that Jepsen found were centered around the "transactions" feature that Cassandra added. We don't use these and don't need them since we don't need 100% consistency and prefer availability (for example we read at quorum to trigger read repair, but downgrade to single node reads if we need to).

Also ScyllaDB is a new product and it would be crazy to start off with it. We plan to run a long-term double write experiment before we are comfortable with using it as a primary data store.

<comment><USER18> The Jepsen tests were not completely centered around transactions. It also had to do with data loss when replicas go down and pure "last-write-wins" approach. For those wanting more info around this the original post is here:

[https://aphyr.com/posts/294-jepsen- cassandra](https://aphyr.com/posts/294-jepsen-cassandra)

<comment><USER3> Last-write-wins is a semantic we are okay with for this data, and dealing with one of our conflicts is outlined the article.

<comment><USER18> I enjoyed your article and I do appreciate your transparency!

</comment></comment></comment></comment></comment><comment><USER19> _> While Cassandra has schemas not unlike a relational database, they are cheap to alter and do not impose any temporary performance impact_

in most relational databases, the schema is cheap to alter and does not impose a temporary performance impact.

In-fact, all of their requirements (aside of linear scalability) could also be met with a relational database. Doing so would gain you much more flexible access to querying for various reports and it would reduce the engineering effort required for retrieval of data as they add more features (relational databases are really good at being queried for arbitrary constraints).

I think people tend to dismiss relational databases a bit too quickly these days.

<comment><USER5> a) I'm not aware of any relational database that can alter a schema in real time on a hot table with billions of records.

b) You were quite okay to just dismiss scalability there except that's the most important requirement for a company such as this. People don't just choose Cassandra lightly given how significant its tradeoffs are.

c) Most companies are offloading analytics/reporting workloads into Hadoop/Spark and then exporting the results back to their EDW. This allows for far more functionality and keeps your primary data store free from adhoc internal workloads.

d) Nobody dismisses relational databases quickly. In almost all cases they are the first choice because they are so well understood. The issue is that most of them do have issues with linear scalability and the cost to support them quite prohibitive e.g. Teradata, Oracle.

<comment><USER20> I sort of agree with threeseed and and the GP comment, so upvotes to both of you.

1) Altering schema is vague. It was used vaguely in the article (although, given the clarity of the article, I suspect the authors knew exactly what they meant). Some alterations on relational database tables are fine, even when hot and have billions of rows. Others are not.

Add a new column: fine. Index the new column: fine. Create a new index on a column with billions of rows: definitely not fine.

But the index plan described in the article was very specific about what they wanted. It doesn't sound like they had to add any new indices.

2) Mostly agree here. Linear scalability is a big deal here, and it's fucking hard to do well for most RDBMS systems. I slightly disagree, however, because the article explicitly states that the requirements are willing to trade C for A in CAP theorem. This is important. The hardest parts of linear scaling in RDBMS are enforcing C. Think transactional systems that absolutely must be consistent. Like your bank account. This isn't that, and the blog post clearly states it. Takes a lot of pressure off the relational database when it comes to scaling.

3) Strongly disagree. Most companies don't have the resources or manpower to do that. It takes a lot of time and a lot of effort. Hell, most companies don't even have an EDW. Let alone a pipeline from the OLTP server to Spark/Hadoop to the non-existent EDW.

4) We seem to run in different circles. Almost everyone I know dismisses relational databases without question. Mongo is the way to go. And I get called out as the resident old fart/luddite who insists on using postgres. Speaking of which, if the first things you think of with relational DBs are Teradata and Oracle, we are definitely operating in different contexts.

If your opinion is that relational databases are generally well understood by --and therefore often the first choice for--developers . . . I want to know where you work.

Because that's not a different context from where I am.

That's a different universe.

The reality is that storing and retrieving data is a hard problem, and there's no set answer that works for everyone in every situation. If you're building a new product from scratch, you should go with what your team knows, provided that the team knows enough to not put yourself in the situation where you're just losing data in a partition scenario (well-made point in the original article. Mongo is fine on one node. Scale it out, and you might as well write your data to /dev/null)

Almost any datastore will serve the needs of a new product until it needs to scale horizontally. Relational, NoSQL, Object store, whatever. When it comes to scaling linearly, you have to take factors into account.

1) Which part of CAP theorem are you willing to sacrifice? You always have to let go of one.

1a) If you want a CP system, you have no choice but to deal with scaling problems of relational databases. You must have transactional guarantees for this to work.

1b) If you need an AP system, you have choices, but the choices lean in favor of systems like cassandra. It's just easier than seting up multi-node postgres and doing sharding.

It's also worth pointing out that people very often dismiss vertical scaling too soon. Take a look at Joel Spolsky's articles about infrastructure at StackOverflow. You can do quite a lot with the available firepower of modern technology by just buying bigger and better hardware.

I'm not suggesting that going bigger would have been the right choice for Discord. But sometimes it can be the right choice.

If there's something I fundamentally disagree with about the article, it's this: trying to do everything in a single data store. I think--much like what you suggested above--that it's better to have separate systems for reading and writing. Since the use case is definitively AP, I can't see a reason not to have a transactional system in an RDBMS and a streaming pipeline to a cassandra cluster for reading.

Use the right tools for the right job, is basically my point.

<comment><USER15> > The hardest parts of linear scaling in RDBMS are enforcing C.

The hardest parts of linear scaling in RDBMS is actually doing the scaling - it's "what do I do when I'm about to outgrow a master and need to add a bunch of capacity", and "what do I do when the master crashes". At Crowdstrike we would add 60-80 servers to a cassandra cluster AT A TIME, no downtime, no application changes, no extra work on our side - just bootstrap them in, they copy their data, and they answer queries. The tooling to do that in an RDBMS world probably exists at FB/Tumblr/YouTube, and almost nowhere else.

> Think transactional systems that absolutely must be consistent. Like your > bank account

Most banks use eventual consistency, with running ledgers reconciled over time.

> It takes a lot of time and a lot of effort. Hell, most companies don't even > have an EDW. Let alone a pipeline from the OLTP server to Spark/Hadoop to > the non-existent EDW.

In the cassandra world, it's incredibly common to setup an extra pseudo- datacenter, which is only queried by analytics platforms (spark et al). Much less work, and doesn't impact OLTP side.

> 1a) If you want a CP system, you have no choice but to deal with scaling > problems of relational databases. You must have transactional guarantees for > this to work.

This is fundamentally untrue - you can query cassandra with ConsistencyLevel:ALL and get strong consistency on every query (and UnavailableException anytime the network is partitioned or a single node is offline). Better still, you can read and write with ConsistencyLevel:Quorum and get strong consistency and still tolerate a single node failure in most common configs.

> Use the right tools for the right job, is basically my point.

And this is the real point, with the caveat that you need to know all the tools in order to choose the right one.

<comment><USER20> The fuck are you even talking about?

1) scaling is easy . . . oh casandra. Where you can't have C and don't care about P.

2) Let me tell you about banks. I used to work for banks. Banks do not use systems that are eventually consistent. Banks use systems--however old and outmoded--that are strongly consistent. Banks do not use systems that are eventually consistent except for ACH transfers. And that's not a database. That's a flat file

3) There is no cassandra world that you speak of. This is utter bullshit.

4) No it's not untrue. Cow--as we call it on me team--absolutely sucks at C when you're talking about scaling horizontally.

Make up your mind. Is this good at single node guarantees or is it good at sharded guarantees?

Pick one.

We know for a fact that if you want CAP, you can't have all three. You can have AP or CP, but you can't have all of them. If you're arguing that you can have C and A, you have failed at P.

Maybe that's a thing you're willing to trade-off. But it doesn't in any way relate to my point.

My point, if you missed it, was this: if you want strong consistency, you need a relational database, and you need transactional guarantees.

That is hard to do, and no one does it well yet. You're just lying to people if you say otherwise.

<comment><USER15> I don't know what your background is, but I'm really encouraged by the fact that I've worked my whole career without having to deal with people that behave like you.

> 1) scaling is easy . . . oh casandra. Where you can't have C and don't care > about P.

This isn't about teaching me the CAP theorem. I know the CAP theorem. I know the tradeoffs. I've built and managed systems that handle hundreds of billions of events a day, writing millions of writes a second into a thousand cassandra nodes. You can have C, if you want it - you dont get transactions with rollbacks, but that doesn't mean you dont have consistency.

> 2) Let me tell you about banks. I used to work for banks. Banks do not use > systems that are eventually consistent

All this time, I thought ING was a bank: [https://www.youtube.com/watch?v=EiqdX23u_Mk](https://www.youtube.com/watch?v=EiqdX23u_Mk)

Also: [http://highscalability.com/blog/2013/5/1/myth-eric-brewer- on...](http://highscalability.com/blog/2013/5/1/myth-eric-brewer-on-why-banks- are-base-not-acid-availability.html)

> 3) There is no cassandra world that you speak of. This is utter bullshit.

I see, lame troll or wholly clueless. Guess I'm done.

</comment></comment></comment></comment></comment></comment><comment><USER21> Love Discord. Most of my friends and I have switched over from using Mumble and it's been great.

I run a small Mumble host [1] and I've always thought of the idea of wrapping the Mumble client and server APIs to function like Discord/Slack as an open source alternative. Mumble is great and all, but the UI/UX appeal of Discord is so much better.

Keep up the great work!

Also, is this is the same Stanislav of Guildwork? Ha, I remember when Guildwork was being formed back in the FFXI days.

[1] [https://guildbit.com](https://guildbit.com)

<comment><USER3> I am :) glad people remember me from FFXI haha

Add me on Discord if you ever wanna chat, Stanislav#7943

</comment></comment><comment><USER15> Wildly biased Cassandra person, but I find this very well written and explained, and I'm especially happy that when you bumped into problems like wide partition and tombstone memory pressure, you didn't just throw up your hands, but you worked around it.

The wide partition memory problem should be fixed in 4.0, for what it's worth.

</comment><comment><USER22> Discord missed an opportunity a year or two ago to become something like slack for large companies. Hipchat's perf is horrible and slack couldn't scale to +20k users a year ago. Managing a mattermost instance requires staff and is more outage prone.

It's really too bad that they didn't take advantage of it, since they were actually scalable compared to their competitors and had good voice chat. Slack has started becoming more scalable recently, so I don't know how much the opportunity is still there.

<comment><USER3> We are larger than Slack

<comment><USER23> In what metrics? Number of users? Number of paying users? Valuation? Headcount?

<comment><USER24> Seconding this question. Can you share the equivalent discord numbers for the slack numbers at the link below?

[http://expandedramblings.com/index.php/slack- statistics/](http://expandedramblings.com/index.php/slack-statistics/)

<comment><USER3> We currently don't share exact metrics for all those stats, but we have shared a few press releases and blog posts which you can easily extrapolate from. :)

</comment></comment></comment></comment></comment><comment><USER25> If you're deleting often, I recommend running a full compact (after your repair) to free up space and rid yourself of those tombstones once and for all. Repairs without compacts make those SSTables grow and grow. It's amazing how much space a compact clears up.

<comment><USER26> I had to delete a shitload of data from Cassandra recently and it required dropping gc_grace_seconds to a very low value in order for the tombstoned records to be dropped during compaction (this was mentioned in the article)

</comment></comment><comment><USER27> Not surprised to see other companies facing issues with Cassandra and tombstones. Don't get me wrong, I understand the need for tombstones in a distributed system like Cassandra... It doesn't make it any less of a pain though :).

<comment><USER15> The tombstone problem described is due to misuse - probably from improper use of prepared statements. Looks like they worked around it well.

<comment><USER28> > The tombstone problem described is due to misuse.

My concerns with Cassandra are precisely here: this is easy to misuse it.

There are a lot of constraints on the schema (more specifically on the design of partition & clustering keys). Each choice leads to many restrictions on what can be requested/updated/removed; and to different issues with tombstones and GC.

The Discord's story is exactly what I experimented: a series of surprises, and even really bad surprises in production. In both cases, the story ended with an efficient system, but with by far more engineering work and rework than initially planned.

<comment><USER29> This article is extremely similar to almost every other "OMG! We used Cassandra and it was nothing like a SQL database!" article by Netflix, Spotify, and so many more. The fact that every single one contains the same 6 or 7 self-inflicted issues is pretty funny to me. I mean, I thought we lived in the age of the Google Dev.

Cassandra does require you to know more of it's internals than most other data stores. Unfortunately, the move to CQL and very SQL-like names for things that are nothing at ALL like their SQL counterparts is not helping.

Also, our own personal death by tombstone: A developer who didn't even know those existed checked in some logic that would write null into a column every time a thing succeeded.

After that passed QA and went into production, all hell broke loose with queries timing out everywhere. SUCH FUN.

</comment></comment></comment></comment><comment><USER30> I'm one of the people who nagged you on the redis post, and particularly expressed skepticism that such a transition would've been necessary. I haven't read this yet, but I just want to say thanks for actually following up to that thread and posting it. Looking forward to it!

\</comment><comment>---

EDIT: Just read the post, and while it provides a good perspective on Discord's rationale to introduce Cassandra in the first place and does a great job pointing out some unexpected pitfalls, it doesn't specifically respond to replacing Redis with Cassandra due to clustering difficulty, per the prior thread. [0] Redis is only specifically called out as something they "didn't want to use", which I guess is probably the most honest answer.

The bucket logic applied to Cassandra seems like it could've been applied to redis + a traditional permanent storage backend nearly as easily. The biggest downside here would be crossing the boundary for cold data, but that's a pretty common thing that we know lots of ways to address, right? And Cassandra effectively has to do the same thing anyway, it just abstracts it away.

Again, I'm left wondering what specific value Cassandra brings to the table that couldn't have been brought by applying equal-or-lesser effort to the system they already had.

I also found it amusing that they're already contemplating the need to transition to a datastore that runs on a non-garbage-collected platform.

[0] [https://news.ycombinator.com/item?id=13368754](https://news.ycombinator.com/item?id=13368754)

<comment><USER3> This post was about using Cassandra for message storage.

You are basically advocating for plugging 2 systems together, which out of the box don't provide elasticity. Or we could just use Cassandra. It is a simpler solution and Cassandra is not new tech. Aside from caching empty bucket information we have nothing sitting in front of Cassandra. It works great and the effort was minimal.

The Redis comment by jhgg was referring to our service which tracks read states for each user. We might write about that later, but it's not as interesting. The most interesting about that experience was reusing memory in Go to avoid thrashing the GC.

We care about seamless elasticity for our services which Redis doesn't provide out of the box except with Redis Cluster which does not seem to be wildly adopted and forces routing to clients.

<comment><USER30> Ah, I see. Thank you for clarifying that this is not the service to which jhgg referred.

Obviously, I haven't addressed this problem in-depth and I don't really know enough about the specifics to criticize the decision directly. It's completely possible that Cassandra was the perfect fit here.

The previous thread was in the context of switching things up without strong technical motivation. I said that actually, it _does_ seem easier to fix a redis deployment than to write a microservice architecture backed by Cassandra, and that I hope to hear more about a stable production ethos from the tech community as a whole. There are a lot "moving on to $Sexy_New_Toy_Z" posts, but not a lot of "we solved a problem that affected scaling our systems, and instead of throwing the whole component away and starting over, we actually did the hard work of fixing and optimizing it: here's how".

To address your specific complaints.

>You are basically advocating for plugging 2 systems together, which out of the box don't provide elasticity.

I mean, again, without getting in-depth, I'm not advocating _anything_ (I feel like I need a "This is not technical advice; if you have problems, consult a local technician" disclaimer :P).

However, storing lots of randomly-accessed messages and maintaining reasonable caches are not new problems. There are lots of potential solutions here.

And while Cassandra is not "new tech" in the JavaScript-framework-is-a- grandpa-after-6-months sense, it's certainly "new tech" in the "I'm the system of record for irreplaceable production data" sense.

Cassandra is also among the less-used of the NoSQL datastores, putting it in a minority of a minority for battle-tested deployments. You mention Netflix and someone other big name using it in production as part of your belief that it's stable. This, I think, is part of the problem.

These big companies use these solutions because

a) they truly do have a scale that makes other solutions untenable (although probably not the case with Cassandra itself);

b) they can easily afford the complex computing and labor resources needed to run, repair, and maintain a large-scale cluster. Such burdens can be onerous on smaller companies (esp. labor cost);

c) when they need a patch or when something starts going awry, they can pay anyone whose willing to make the patch, their own team not excluded. Often the BDFLs/founders of such projects end up _directly employed_ by these big companies that adopt their tech.

"Netflix [or any other big tech name] uses it so we know it's stable" is a giant misnomer, IMO.

None of this is to say that Cassandra isn't a good choice for this problem or any other specific problem, because again, as a drive-by third-party I don't know. But contrary to what the article states, it hardly seems like Cassandra was the only thing that could've possibly fit the bill. I bet it could be done well with a traditional SQL databases (which, from the body of the post that identifies Discord as beginning on MongoDB and planning to move to something Cassandra-ish later on, it doesn't sound like was ever tried or considered).

It's kind of like reading an RFP that was written by a a guy at a government agency that already knew they really wanted to hire their brother-in-law's firm. "Must have $EXTREMELY_SPECIFIC_FEATURE_X, because, well, we must! And it just so happens that this specific formulation can only be provided by $BIL_FIRM. What d'ya know."

>We care about seamless elasticity for our services which Redis doesn't provide out of the box except with Redis Cluster which does not seem to be wildly adopted and forces routing to clients.

First, you just admitted that Redis _actually does_ have that feature that you're saying it doesn't have. "Redis Cluster" and "Redis" are the same thing. "Redis Cluster" is part of normal redis and afaik, while it requires additional configuration, it will automatically shard.

In any case, while I have no numbers, I would wager that Redis Cluster is more widely used than Cassandra.

<comment><USER6> Cassandra doesn't require all the data to live in RAM. At this scale, you need disk-based data access.

</comment></comment></comment></comment><comment><USER31> Serious question, how do you backup a casandra database of that size. Do you even back it up or just rely on the sharing to prevent dataloss?

<comment><USER3> Cassandra has a snapshat command that creates a directory by symlinking files that hold data (this is safe cause Cassandra files are immutable). Then you just upload them to your backup storage. This is obviously for recovery scenarios that are catastrophic.

Normally though since the data is replicated on 3 nodes, you can technically loose a node completely and rebuild it from the other nodes.

</comment></comment><comment><USER32> I love Discord and use it on a daily basis, one of our main concern with my gaming group is the voice latency compared to TS, Mumble or Ventrilo but this is mainly due to the inability to host your own server.

One of the big missing feature we would like to have in Discord is the ability to assign special permission to our groups leader so they can communicate over voice chat to other other group leaders in other channels (global voice chat).

When we play PVP MMO's and have 40+ users all in the same channel calling shots its impossible to coordinate properly.

What we normally do is split the group in 4 so 10 players in 4 different channels and each group leaders are calling shots independently BUT can also communicate via voice chat to other group leaders. Basically there's a global voice chat for group leaders that no one else can hear but them.

Other than that Discord is amazing!

</comment><comment><USER33> For a bit more information about the tombstone issue from the perspective of the person who caused it: [https://www.reddit.com/r/programming/comments/5oynbu/_/dcnxy...](https://www.reddit.com/r/programming/comments/5oynbu/_/dcnxy5s)

</comment><comment><USER34> I'm curious why would people use a closed source software, when you can use something like [https://riot.im](https://riot.im)

Please let me know. I may be missing something.

<comment><USER35> Games vs general use. By targeting gamers Discord is able to customize the experience to make it better for games.

<comment><USER36> Kind of like Twitch vs Youtube even

<comment><USER35> Exactly.

</comment></comment></comment></comment><comment><USER37> Discord is great but I have intermittent performance issues with it that make it almost unusable in comparison to Slack which never has any noticeable latency.

<comment><USER3> This should not be the case, we strive for Discord be lightning fast at all times.

Please email me email@example.com I would love more details.

<comment><USER37> I can't really give you "more details," I was trying to use Discord with some people and every couple minutes or so the chat would freeze up and then flood with messages. It wasn't just me, it was everyone in the room's Discord doing it, and it really doesn't seem like a client-side bug.

</comment></comment></comment><comment><USER38> > Having a large partition also means the data in it cannot be distributed > around the cluster.

Why can't a large partition be distributed around the cluster?

<comment><USER6> Cassandra uses consistent hashing. A partition is a segment of data identified by the partition key to determine which node in the consistent hash ring owns that data.

You cant break down partitions any further because it's just a name for the smallest cohesive set of data owned by a hash key, so instead it's advisable to use more partitions with data modeling rather than making them huge.

</comment></comment><comment><USER39> Does anyone know what protocol/transport Discord uses? XMPP, web sockets, JSON, etc?

<comment><USER40> Our realtime messaging is done over Websockets using either JSON (for web/non- native) or ETF ([http://erlang.org/doc/apps/erts/erl_ext_dist.html](http://erlang.org/doc/apps/erts/erl_ext_dist.html)). Almost all user-based actions are sent over our HTTP API using JSON.

<comment><USER41> Since Cassandra is eventually consistent, how do clients get a consistent sequence of messages?

Do you actually use Cassandra range queries to poll for new messages, or do clients use some kind of queue to get notified?

Say messages A, B, C are created in that sequence. But isn't it then possible that a client asking for new messages only gets A and C, and B only shows up a few milliseconds later, which would be missed unless the client actually asked for "everything since before A"? Or is that not possible?

<comment><AUTHOR> We have a real time distribution system written in Erlang/Elixir that keeps the clients in sync. We do not poll for messages. That wouldn't work at our scale :)

</comment></comment></comment></comment><comment><USER42> What's an upsert?

<comment><USER3> Update if exists, otherwise insert.

</comment></comment><comment><USER43> You're storing messages, how are you guaranteeing safety of those messages when it looks like one can seemingly just blast through your API calls to find messages when one isn't even on that server?

<comment><USER44> A few seconds of research would have revealed that their API requires an authentication token[1]

[1][https://discordapp.com/developers/docs/reference](https://discordapp.com/developers/docs/reference)

<comment><USER43> A few seconds of pen-testing tells me their token revocation is pretty shoddy. I'm now spying on my fiance's Discord chat.

Bravo to those that downvoted me without even bothering to use their brains (let alone test something.)

<comment><USER3> The normal token system revokes on password change, if you want to revoke and have extra security we offer MFA login which has unique tokens per login. If security is of importance to you then use MFA.

<comment><USER43> Why are you not revoking tokens after session end across the board? Token re- use is one of the faster-rising security breach factors now days.

</comment></comment></comment></comment></comment><comment><USER45> Wow! This is an incredible article. I do research and development for systems like this at GUN, and this article nails a lot of important pieces. Particularly there ability to jump to an old message quickly.

We built a prototype of a similar system that handled 100M+ messages a day for about $10, 2 minute screen cast here: [https://www.youtube.com/watch?v=x_WqBuEA7s8](https://www.youtube.com/watch?v=x_WqBuEA7s8) . However, this was without FTS or Mentions tagging, so I want to explore some thoughts here:

1\. The bucketing approach is what we did as well, it is quite effective. However, warning to outsiders, this only effective for append-only data (like chat apps, twitter, etc.) and not good for data that gets a lot of recurring updates.

2\. The more indices you add, the more expensive it gets. If you are getting a 100M+ messages a day, and you then want to update the FTS index and mentions index (user messages' index, hashtag index, etc.) you'll be doing significantly more writes. And you'll notice that those writes are updates to an index - this is the gotcha and will increase your cost.

3\. Our system by default backs up / replicates to S3, which is something they mention they want to perhaps do in the future. This has huge perks to it, including price reductions, fault tolerance, and less DevOps - which is something they (and you) should value!

There backend team is amazingly small. These guys and gals seem exceptionally talented and making smart decisions. I'm looking forward to the future post on FTS!</comment>


How to deal with bad stereotypes at meetings? - <AUTHOR> https://blog.weekdone.com/how-to-deal-with-12-bad-stereotypes-at-meetings/

<comment><USER2> Who has not seen any of those guys at their meetings?</comment>


Megaupload Wins Crucial Evidence Disclosure Battle With US Govt. - <AUTHOR> http://torrentfreak.com/megaupload-wins-crucial-evidence-disclosure-battle-with-us-govt-120529/

<comment><USER2> This is amazing news. Everyone involved in this case knows that the US overstepped its boundaries on this one, and once the case is thrown out of court expect there to be a shitstorm as Dotcom files for damages and lost revenue from the site. I don't know how much Megaupload made per month but I know it was a pretty astronomical figure, times that by the number of months Megaupload has been forcefully offline and it'll be a large ending figure (that's without factoring in growth factors).

I really do hope this whole situation exposes the inner workings of the heavily corrupted US judicial system. It's ridiculous the FBI is involved in what everyone knows is a civil matter dressed up in made up criminal charges.

I would really hate to be the recording industry and US government right about now.</comment>


Making a job offer? Lowest isn't always best. - <AUTHOR>

http://www.supercoders.com.au/blog/makingajobofferlowestisntalwaysbest.shtml <comment><USER2> In my experience, most employers start with a lower offer under the assumption that the candidate will want to negotiate. If the employer starts with their best offer and the candidate wants to negotiate they have no room to increase the offer and see this as being a negative. I suppose they want to be able to counter during the negotiation and seem responsive.

Personally, I always focus on making the best possible offer first and if the employee wants to negotiate telling them it's the best offer. It shows the potential employee that you respect them and have spent time considering the offer, instead of just floating something to see how they react.

If you hold back, aren't you just telling the candidate that you were trying to pull a fast one on them if you increase the offer later?

<comment><AUTHOR> I do agree. I reckon put forward the best job offer you can as the first offer.

</comment></comment><comment><AUTHOR> Clickable [http://www.supercoders.com.au/blog/makingajobofferlowestisnt...](http://www.supercoders.com.au/blog/makingajobofferlowestisntalwaysbest.shtml)

</comment><comment><USER3> This isn't how you submit a story on HN...

<comment><AUTHOR> Hello I see from your great karma that you could better advise me. How can I do better?

<comment><USER3> You enter the URL in the URL input not the text box.

</comment></comment></comment><comment><USER4> we have pretty much a fixed salary for each level. If someone wants more salary, then we sometimes will give them the next level (or just say no). If they arent at that level, then they need to very quickly get the skills to be executing at the level.</comment>


Business Monkey Wants to Learn Programming: - <AUTHOR>

I have been reading Hacker News for a few months and noticed the posters are primarily technical in nature.
I have limited experience with VB and I have decided to learn Ruby on Rails.
I have set out to make a web based application which allows the user to enter a single piece of data and hit submit which then puts the data in a secure database. (I figure this will be a challenge, but something attainable and a good solid start to my programming.)
Does anyone have any suggestions on the best way to accomplish this?  So far, I have installed Ruby on Rails. <comment><USER2> I am pretty much doing the same thing, and I've found good success using Ruby Koans, Rails for Zombies and then Rails Tutorial as dustinchilson points out.

The order helped me out though. I started with Ruby Koans which taught me the fundamentals of Ruby and let me make a program or two without needing to setup a dev computer...it's an easy way to just get a feel for programming and learn about why Ruby works in the way it does.

<http://rubykoans.com/>

Next I moved onto Rails for Zombies. This was a great, great site because again you could learn the fundamentals of rails with a lot of tutorial help in a completely online browser based environment.

It really helps you get the basics of creating a typical application and it's fun too.

<http://railsforzombies.org>

And finally, I moved on to Rails Tutorial which is amazing because you actually learn how to setup your computer to program and deploy code (which I think is harder than writing it at the moment)!

It goes from zero to deploy and you write a similar app as the one you will see in Rails for Zombies, but this time from complete scratch. Which is an amazing lesson that I can't stress enough.

<http://ruby.railstutorial.org>

My recommendation is in that order because it allows for quick wins, overall understanding (I get MVC now!), and a meaty project that you can feel proud of at the end.

It's grueling at times to learn, but well worth it. Good luck and try to stay dedicated and focused. (I've been wavering a bit, but I'll figure it out).

</comment><comment><USER3> To start getting used to the ruby language I recommend using <http://tryruby.com>.

For rails there are quite a few tutorial sites. Here are a few good ones. <http://railsforzombies.org/> <http://edgeguides.rubyonrails.org/getting_started.html> <http://ruby.railstutorial.org/ruby-on-rails-tutorial-book>

If you go through some of these they will talk you through creating a form and submitting the data to a database. Your application is suite similar to the typical blog application many tutorials take you through building.

To secure the data stored in the database just think of the data as a password and store it encrypted using a salt.

Good Luck.</comment>