Substack has spawned a new class of newsletter entrepreneurs - <AUTHOR> https://digiday.com/media/how-substack-has-spawned-a-new-class-of-newsletter-entrepreneurs/ <comment><USER2> Substack has been successful in creating buzz among the media crowd, and there's definitely a business there riding the trend toward subscriptions in the media biz, but unfortunately taking on $17M in VC money has set them up for a disappointing end.

Here's how this plays out. The successful writers on Substack will eventually leave the platform for more control over their audience and lower fees (there's a reason Ben Thompson isn't on Substack).

As this happens, and as the honeymoon PR period dies off, the journalists will turn on Substack, just as they do on every single thing Silicon Valley puts out (see Lambda School).

Prior to taking on a16z riches, Substack was a cool indie brand. Now they're officially "big, evil tech." The fickle journalist twitterati just haven't realized it yet.

Inevitably as more non-influencer writers head to Substack they'll start to see, like with Patreon, that only a small minority of creators will ever generate enough income to make a living.

This, combined with competition from Patreon, Podia, Memberful, Ghost, Revue, the next indie platform darling, etc. will spell doom for a16z's growth goals for the platform.

My guess is 3-7 years from now, after more funding rounds, they get acquired by Patreon for roughly the same amount of money investors put in. Employees will get nothing, founders might get Cush jobs and possibly a minor payout, VCs would have done better investing in the S&P.

This is the type of business that works better bootstrapped IMO.

<comment><USER3> > Here's how this plays out. The successful writers on Substack will > eventually leave the platform for more control over their audience and lower > fees (there's a reason Ben Thompson isn't on Substack).

Except the entire model of substack is based on writers not having access even to their audience's mail addresses.

<comment><USER4> Substack writers do have access to their signups and can easily export them.

[https://support.substack.com/hc/en- us/articles/360037465992-...](https://support.substack.com/hc/en- us/articles/360037465992-How-do-I-export-my-email-or-subscriber-list-)

<comment> edouard-harris Yes. This has been the case from the very beginning of their platform, as far as I'm aware.

</comment></comment></comment></comment><comment><USER5> Maybe this varries depending on interest niche, but for me personally, there is so much quality writing that is available for free, much more than i am reasonably able to consume, its hard to imagine paying a subscription for writing.

<comment><USER6> You might pay for the curation. Finding what's interesting takes time as well, even if it's just scrolling through HN, and you will miss some things. If somebody saves you that time, that's probably well worth a few bucks a month, isn't it?

<comment><USER7> If it was a one-stop shop that fed me all the reading i could do, i would happily pay £10 a month for it. Maybe a bit more.

But if it's narrowly focused, the value is proportionally less. If someone does really good curation of obscure Soviet aerospace technology, i'm interested, but that's no more than 5% of my reading.

But then, i am a generalist, and fairly stingy. It seems like a repeat of a very old story to say that niche curators will have to find either price- insensitive generalists ("whales"), or focused obsessives.

</comment></comment></comment><comment><USER8> I think what Substack is doing is really exciting, but it remains to be seen how large this phenomenon will become. I heard on a CNBC interview that there are only about 30 writers who make a living income from Substack right now (don't know if it's true or not).

"Pay money to support the creator" can work - Patreon is doing around $50M in revenue annually. Bundling can obviously also work with many big and established examples. But it strikes me that the two models attract a very different customer, so those who subscribe to a specific Substack newsletter may or may not be enthusiastic about buying a big bundle of them down the line.

</comment><comment><USER9> I was going to read the article, but if you think covering 1/3 of my screen with an uncloseable "you've read X articles" overlay is sensible, then I think not reading your article is sensible.

</comment><comment><USER10> One of the founders here. I'd love it if some of the HN crowd applied for our fellowship: [https://on.substack.com/p/announcing-the-next-substack- fello...](https://on.substack.com/p/announcing-the-next-substack-fellowship)

<comment><USER11> Does "anywhere in the world" include countries where Stripe isn't available?

I've toyed around with the idea of creating a Substack, but I can't get paid over Stripe.

</comment></comment><comment><USER3> In the same sense Uber has spawned a new class of driving entrepreneurs.

The only asset a newsletter author has is his audience - with substack they won't even have that, in fact their publisher agreement explicitly prohibits the authors from asking any sort of contact information from their readers.

</comment><comment><USER12> In bigger picture, this is a natural move from quantity to quality. The free content on internet feel like eating garbage. I follow a few really good newsletters (e.g Craig Mod's) and every episode feels like a chapter in a good book.

What I don't enjoy about newsletters is the e-mail clients themselves. They're bulky, filled with features I never use. I wish there was a way to hook my newsletters subscriptions up with Kindle. I can customize the fonts, sizes, spacing for my best reading experience, and the writer can forget about styling and just can focus on the writing.

Which makes me fantasize myself creating a subscription based reading platform with its own device :)

<comment><USER13> It is very easy to convert HTML to ePub, you could create simple script that converts every newsletter into ePub and adds it to your library. I don't know if it would work for Kindle devices, but you could always install custom firmware that has support for Dropbox synchronisation.

<comment><USER11> ePub in general doesn't work on a Kindle (you have to convert them to mobi), but you can create your own "Kindle magazines" (don't know their exact name) using calibre.

One use case of that is that you can feed RSS feeds to your calibre's news feature and schedule them to be sent to your @kindle.com email address at a specified day / time of the week. Essentially DYI magazine filled with whatever blogs you want.

<comment><USER13> That sounds nice! I have a Kobo e-reader, and I use their beta browser or pocket to read this kind of things, but especially the browser is not very convenient.

<comment><USER11> I've switched from a Kindle to Kobo a few months ago, and it's one of the rare features that I miss. Pocket integration gets me close enough, but Kindle really nailed the magazine approach in my opinion.

On the bright side, Kobo at least makes a distinction between a book and an article, while Kindle just treats everything as a book.

</comment></comment></comment></comment></comment><comment><USER14> I am happy to pay for quality writing I can't find elsewhere, and in fact I subscribe to a Substack.

I don't necessarily see the nascent bundling as a problem. Newspapers, which were the workhorses of writing, were bundled geographically which makes less sense now. Bundling by interest makes a lot more sense.

Hopefully the writing supported by this paid model can be as creative as cable TV used to be, compared to network TV.

</comment><comment><USER15> Can those people actually make a living wage from it, or are they "entrepreneurs" like all those Instagram "influenzas"?

<comment><USER16> Some people are (book) authors professionally, some people are authors recreationally, some people are authors professionally _exclusively_ , and some people are authors professionally but their primary monetization engine is elsewhere. If one believes that writing books generally isn't a sustainable occupation, I think that belief is likely under-nuanced.

(Ditto e-books or similar information products, by the way.)

Turning to the question of paid newsletters: I'm socially constrained from pointing at examples, both because in some cases I have non-public information and in some cases I would be perceived as having non-public information, and because there are heady professional considerations, but: there are definitely, definitely some people who sell information professionally who break the top end of the pay scale for people who are generally described as writers in more traditional media jobs.

Another way to put it is "I'm pretty sure I could sell 1,000 subscriptions at $20 a month within 6 weeks if I wanted that to be my job", which is not competitive with other things to do with my time but probably clears most reasonable bars for stable professional options. I think I'm very far from having the largest collection of people who'd pay to read my writing, the best writing, the most engaged userbase, the most lucrative decision points that users routinely apply my advice against, etc.

A big open question for the business model is whether one thinks there are hundreds, thousands, tens of thousands, or hundreds of thousands of people who could hit that bar (and, I suppose, whether one could tempt them from other options). Hundreds feels clearly wrong to me as an upper bound.

<comment><USER17> Thanks for a more nuanced take on the model. The question in my mind is: How much time/energy gets eaten up in marketing a newsletter (or other paid subscription information product)?

I suspect that lack of interest or skill in marketing is much more the limiting factor that sets an upper bound on how many people can pull it off to derive a realistic income this way.

<comment><USER16> I'd imagine it is different for authors in different phases of the business, starting at very substantial while one does not have a lot of a personal platform and decreasing markedly when one does.

Someone, and I'm forgetting who it was, suggested that paid subscriptions is the natural monetization strategy for people who are a) public intellectuals and b) do a lot of their work on Twitter, and in that sense you're doing marketing with a sort of cycles which might not read as "intentionally doing marketing work." (I think as the community distributes the technology of running these businesses, like the technology of running e-book businesses has been distributed, this will get easier, too. One thing all newsletter authors should be doing ~constantly is hitting up podcasts in adjacent spaces, because it gets juices flowing for writing and because every time you talk to 10,000 people you'll sell N new subscriptions.)

</comment></comment></comment></comment><comment><USER18> It's nice to see a different business model of writing. Medium uses subscription model for a whole while Substack uses subscription model for each author.

Time will tell which one will work better.

I already subscribed to writers on Substack (all of them write about blockchain/DeFi).

</comment><comment><USER19> Still no custom domains, so you don't really own your audience or your content on Substack.

<comment><USER20> If you want your own custom domain with membership features, then Ghost is probably your best bet. It has built-in member features, open-source and can also be hosted on their Pro platform. They also don't take a cut from your memberships.

<comment><USER19> Lots of different solutions to this (I use Mailgun and/or Migadu for various usecases). Just surprised that such a hyped product gets to be this popular without support custom domains.

Haven't we learned anything from Medium?

</comment></comment></comment><comment><USER21> I believe that platform works the best if you have a niche based newsletter. And that's what makes it best.

</comment><comment><USER22> I tend to think that Substack isn't that new or different from other platforms that tried to reinvent publishing. They basically took an existing mode of publishing - newsletters - and have build a centralized platform that allows individual writers to share their content with their audience.

The difference is the offering towards the audience. What are readers willing to pay for? Content published on a blog or in a newsletter isn't inherently different: it's short pieces conveying thought, opinions, observations,... So, why would anyone be more willing pay for newsletter content rather then content posted on the Web?

There's the simple convenience of having content you subscribed to delivered to your inbox, rather then having to invest time and effort scouring/curating it yourself from the Web (bookmarks, closed platforms where you need to register, etc.). Content just lands at regular intervals in your mailbox, end of story.

There could be a perception that your Inbox is your own personal space as opposed to the Web as this foreign place with inherent dangers to which you have to expose yourself just to get to the good bits. Authors to newsletters become your guests, whereas you are their guest when you visit their websites. So, I think a sense of agency and being in control are big drivers here.

The sense of exclusivity of being part of a fledgling tribe of readers to a small publisher who knows how to appeal to that sense. Receiving an e-mail in your Inbox is as different from visiting a website, as is receiving a personally addressed snail mail is from going out and reading newspapers in the library.

I'm also curious as to how the e-mail / newsletter model impacts Substack's operations. Unlike 'traditional' social media, millions of visitors don't flock a central platform 24/7/356 which comes with hard infrastructure problems. I'm well aware that there will be different challenges that need to be met, but pushing content to individual mailboxes is, in essence, a hybrid form of decentralization and distributed publishing combined with a centralized platform for curation and subscription.

Another assumption I'm going to make is that revenue isn't spread evenly across all publishers on the platform. I suppose it's more of a long tail with a typical Pareto kind of distribution. 20% of the writers account for 80% of the revenue and then there's a sharp drop off. Thing is, I suspect inactive publishers don't consume much if any resources: e-mails in an Inbox have a rather short half-life as opposed to the (perceived) need of keeping content available over HTTP which brings additional running costs with little to no added value.

It's interesting to note that Substack isn't the only offering in this space. A direct competitor came to my mind: [https://www.getrevue.co/](https://www.getrevue.co/)

Personally, given the pre-dominant promotional nature of newsletters, I'm pleasantly surprised how this mode of writing/authoring has been adopted, once again, by budding writers and publishers. After all, journalling via mail has a long historical pedigree. It's great to see how the analogue concept finds traction in the digital age as well.

</comment><comment><USER23> Newsletters are the easiest way to retain readers since it just pops in their inbox -- if the writing is good and/or specialized. Good design also really helps. When I managed the newsletters for a New York urban planning nonprofit, we averaged 23% open rates. Another good hook is when issues don't have a public web archive, or it's paywalled.

I saw the bundling mentioned in the Digiday article with the "Everything" brand, and I unsubscribed since I just wanted the business analysis and not other topics (like productivity advice) mixed in. Maybe bundling is okay for other readers.

I have to check out what Substack offers in terms of analytics and email design templates. Personally, I'm interested in series-based newsletters for storytelling that's pubdate agnostic (everyone gets the same first issue). I'm going to look closer at ConvertKit and BareMetrics.

<comment><USER24> I'm curious, how do you measure open rates for a newsletter? Many email clients prevent external content from loading as a security measure, thus precluding any tracking. Do you adjust the final figure to account for resulting underreporting? Or do you reckon more clients load external content than I think?

<comment><USER23> Open rates were measured (probably) from either track-pixels as well as any normal images hosted by the email provider. When an email is variably considered 'trusted'/not-spam by the client, it'll show images. But even if no images are shown, a link clickthrough in the email (to the email host and redirected to source) would then also be added to the open rate.</comment></comment></comment>


Flappy Tube – Flappy Bird Flipped - <AUTHOR> http://flynngao.github.io/FlappyTube/

<comment><USER2> Thank you. I used 6 hour to make this game when I was playing PS3

</comment><comment><USER3> I'm loving the internet remix culture. This is hilarious.

</comment><comment><USER4> 123</comment>


Ask HN: Why am I feeling dumb with PHP? - <AUTHOR> Preface: pure opinion based, no rant, no hate, no everything.
I started my actual job 2 years ago, so i'm basically using PHP every day. I'm pretty good at it, but sometimes i feel that PHP is making me very dumb about programming.
I feel like i have to deal with tons of quirks and nonsenses, and i feel completely powerless when i have to think how to program. I feel like i have to think twice before solving a problem that in C i would solve in a more "natural" way.
Sometimes i have to touch some Javascript too and it makes me feel the same way, plus async stuff to me is so unnatural that i hate the async concept itself.
Do you have some similar experiences about X programming language? Let me know :) <comment><USER2> Like it or not, async is the future.

If AJAX was not asynchronous your browser would hang up every time some bro put an AJAX call in their web site and it was slow to reply.

If you had somebody developing exploits for memory errors in the C code you write, C would make you feel dumb too.

Long term when we talk about "mixed initiative" systems where some things happen immediately and other things happen on a human timescale that is going to be very async and will require the development of new tools or the rehabilitation of old tools for it to become mainstream.</comment>


Show HN: GPU-Accelerated Digit Recognition with WebGL - <AUTHOR> https://erkaman.github.io/regl-cnn/src/demo <comment><AUTHOR> The demo uses WebGL, and if you can't get the demo to work, you can here([https://github.com/Erkaman/regl-cnn](https://github.com/Erkaman/regl- cnn)) find a recorded gif that shows what it is supposed to look like.

This demo does handwritten digit recognition by evaluating a Convolutional Neural Network on the GPU with WebGL. The network was trained in TensorFlow by this script here([https://github.com/Erkaman/regl-cnn/blob/gh- pages/scripts/cr...](https://github.com/Erkaman/regl-cnn/blob/gh- pages/scripts/create_cnn.py)), and the network was then reimplemented on the GPU by hand with WebGL. The main purpose of the demo was to demonstate how our WebGL framework regl([https://github.com/mikolalysenko/regl](https://github.com/mikolalysenko/regl)) can be used to greatly simplify GPGPU programming in WebGL. The secondary purpose was to test whether evaluating Deep Learning networks in WebGL is doable. To our knowledge(but we may be wrong!), our implementation is the first implementation ever to attempt GPU accelerating neural networks with WebGL And we hope that this implementation will provide a foundation for people who, like us, wish to experiment with Deep Learning and WebGL The GPU implementation can be found here([https://github.com/Erkaman/regl-cnn/blob/gh- pages/src/gpu.js](https://github.com/Erkaman/regl-cnn/blob/gh- pages/src/gpu.js))

Note that this network will probably be slower than the corresponding network implemented on the CPU. This is because of the overhead associated with transferring data to and from the GPU. But in the future we will attempt implementing more complex networks in the browser, such as Neural Style([https://arxiv.org/pdf/1508.06576v2.pdf](https://arxiv.org/pdf/1508.06576v2.pdf)), and then we think that we will see a significant speedup compared to the CPU.

Lastly, if anyone has any questions, I will be glad to answer them here.

<comment><USER2> Nice work! I've also tried my hands at use WebGL to do deep learning in the browser (e.g. [https://github.com/scienceai/neocortex/blob/master/src/lib/w...](https://github.com/scienceai/neocortex/blob/master/src/lib/webgl/matmul- webgl.js)). The conclusion I came to was that there are just way too many limitations for it to really pay off. The need to encode everything in textures etc limits the data shape and dimensionality, not to mention the complexity cost. If you can get more complex networks working I'll be really impressed!

MXnet.js [1] is an emscripten port of the base C++ framework. It runs entirely in the browser and works fairly well. The actual code produced by emscripten isn't that large, but the model weights can become an issue. I've tried to get emscripten working on tensorflow, even just for forward prediction, but have been pretty much gotten nowhere. Of course this doesn't let you harness GPU power.

Lots of cool potential applications of doing deep learning over the web are just waiting to be discovered and built.

[1] [https://github.com/dmlc/mxnet.js](https://github.com/dmlc/mxnet.js)

<comment><AUTHOR> Wow, so there are other people who have tried doing deep learning in WebGL! But I will also give it a try, and see if I can do it better than you.

</comment></comment></comment><comment><USER3> When I handwrite a 7, it has a dash through the middle. Everyone I know does this. Your app doesn't like these; it sees a 3.

<comment><USER4> Really?? It is the opposite here. No one really uses the dash. I like wee differences like this.

<comment><USER5> > It is the opposite here. No one really uses the dash.

Who is no one? Where is here?

<comment><USER6> In my experience the dash is unusual (but not unheard of) in the US. I've heard people refer to it as a German or European 7. I've never seen the German 1 that is like an upside down V used by an American.

</comment></comment></comment></comment><comment><USER7> I keep getting incorrect results for 4's in this style: [http://imgur.com/BIlhnN9](http://imgur.com/BIlhnN9)

However if I draw 4's like this: [http://imgur.com/akifdRs](http://imgur.com/akifdRs) I get the correct result.

Is this a limitation with the training set?

<comment><AUTHOR> that kind of 4 doesn't exist in the training set. This works: [http://imgur.com/gallery/xnFnF](http://imgur.com/gallery/xnFnF)

</comment></comment><comment><USER8> Interesting it returns the false numbers when written in 'non-American' style. Drawing a 1 is recognized as 7, and drawing a 7 (with 'strike-through') is recognized as 2 :)

([https://en.wikipedia.org/wiki/Regional_handwriting_variation...](https://en.wikipedia.org/wiki/Regional_handwriting_variation#Arabic_numerals))

</comment><comment><USER9> Since the network has already been trained, I can't imagine using WebGL is worthwhile. It would interesting to see, either, drawing samples from the equilibrium distribution or training in the browser.

</comment><comment><USER10> It must have been quite painstaking to hand-code this neural network into WebGL shaders. It could have been easier if the browser vendors would just implement the WebCL standard.

This seems like a throwback to the pre-CUDA "GPGPU" era, when people were implementing numerical algorithms in OpenGL to be able to leverage GPUs for general purpose computing.

<comment><AUTHOR> Yeah, it was really painful. In order to speed up the process, I first implemented the network on the CPU, so that I could quickly verify my GPU implementation.

</comment></comment><comment><USER11> I draw my 1's the way it's often displayed on screen, with a little tail at the top and a line at the base. Your system keeps detecting this as a 2. I've also had 3's detected as 2's.

<comment><AUTHOR> Yeah, the MNIST dataset is not that great it seems. At least for recognising real handwritten digits.

<comment><USER11> That's bad news for me, because I also plan to use the MNIST data in a project of mine. :)

</comment></comment></comment><comment><USER12> It may have some issues with 9, depending on how you write it. [https://i.imgur.com/URYuOSO.png](https://i.imgur.com/URYuOSO.png)

</comment><comment><USER5> Pretty cool. I realize it is mainly a proof-of-concept, but decided to try out variations of scribbles[1]. Does the code make explicit assumptions about orientation, or did the training data make certain assumptions?

[1] [http://imgur.com/a/HCFzy](http://imgur.com/a/HCFzy)

<comment><AUTHOR> The original MNIST training data made the assumption that the digits are not flipped. But you could solve it by creating more training data by flipping the original digits. But then you suddenly end up with an awful lot of data, and then the training process would take days, literally.

</comment></comment><comment><USER13> Drew 8s skewed slightly toward left, ended up with "6" or "3"

</comment><comment><USER14> Intriguing! I was under the illusion that webGL was emit-only, and that Javascript programs can't read back any output generated by WebGL. I must be wrong or out of date :) So how does the script do it?

<comment><USER15> With the right extensions, you can render to a texture which can be sampled from in subsequent draw calls.

The framebuffer object extensions that allow you to write to 32-bit RGBA textures are widely supported on desktops and mobiles (OpenGL ES 2). But floating point textures are not. So, shaders resort to encoding 32-bit floats in RGBA textures. This unfortunately isn't a simple cast. More here: [http://aras-p.info/blog/2009/07/30/encoding-floats-to- rgba-t...](http://aras-p.info/blog/2009/07/30/encoding-floats-to-rgba-the- final/)

</comment></comment><comment><USER16> I have trouble with the 9's being returned as 3's

<comment><USER17> same here - I thought for a second that my handwriting was THAT bad after years with keyboard.

</comment></comment><comment><USER18> Very nice! For some reason I thought it was for letters, and was wondering why it thought my 'h' looked like a '6' heh

</comment><comment><USER19> I've never gotten a correct result.

7 returned 9, 3 returned 8, 1 returned 9...

<comment><USER20> 666 returns 4

<comment><USER21> When you break the contract of a function (a single digit here), you might as well be playing dice. [https://xkcd.com/221/](https://xkcd.com/221/)

</comment></comment></comment><comment><USER22> Might be helpful used in a Captcha-type setup

<comment><USER23> What are you envisioning? It would seem that drawing digits is fairly easy for bots.

<comment><USER24> One way would be to _crack_ captchas, eg make a Chrome extension that automatically fills them up. I remember someone implemented that at some point for MegaUpload.

Another way is to gather a data set of people writing digits with their mouse, and make a classifier that tells you if an input is realistic or not. Of course you'd need to store previous user inputs to make sure someone is not just reusing the same digit over and over again.

</comment></comment></comment><comment><USER25> I have tried this. It is always returning 0.

<comment><AUTHOR> Hmm, may be a problem with your graphics card. What's your graphics card and browser?

<comment><USER26> Getting the same issue with chrome 52, windows 10 and a r9 fury.

[.Offscreen-For-WebGL-043C4318]GL ERROR :GL_INVALID_OPERATION : glReadPixels: demo:1

Issue with amd cards maybe?

<comment><AUTHOR> Yeah, reading from floating point textures with `glReadPixels` is not really supported on some cards or browsers, it seems.

<comment><USER26> I just checked chrome://gpu and it shows:

Canvas: Software only, hardware acceleration unavailable

and some other 'Problems detected' so the problem is probably on my side.

Cool project though!</comment></comment></comment></comment></comment>


A flipping fortune: get snapped up for millions - <AUTHOR> http://www.theage.com.au/small-business/startup/a-flipping-fortune-get-snapped-up-for-millions-20121128-2ad7a.html <comment><USER2> This made me smile: Step 1. Build a dream product like Dropbox or Instagram</comment>


Not ad, Up to 200 books free for you, as i will be gone - <AUTHOR> These books include:
30 more Science fiction world (China's most famous scifi magzine, monthly version and mostly translation version).
2 Science fiction world's The 30 anniversary of the supplement.
35 more world classical literatures.
30 more chinese classical literatures.
10 more world famous scifi novels.
some poem, prose.
I have no exactly quantity and category now, may more or less, i will count it this or next week.
If you like book, especially like scifi or classical literatures, and like paper books, you can email me.
BUT almost all the books is in chinese, and you (or your friend) need in BeiJing, you know, books are so heavy.
I have in BeiJing 7 years, no lucky to meet a man love scifi or literatures the same with me, so books is my friends.
They are my friends, but i can't take them away anymore, if you are coming to take them, you will do me a favor, thank you.
firstname.lastname@example.com <comment><USER2> Sometimes there is a little quiet poignant moment on HN - may you be lucky in finding a man to share new books with.

<comment><AUTHOR> Thank you, a man had contacted me, but he is in America, it is too far to send 200 paper books to.

</comment></comment><comment><USER3> where are you going?

<comment><AUTHOR> back to my hometown JiangXi, may be in the end of the year.

<comment><USER4> Glad to hear it, from the title I thought you meant that you would be gone from this earth, and not as a taikonaut.

<comment><AUTHOR> Haha, you are humour, just make 1 billion chinese Laugh.</comment></comment></comment></comment>


The most important revenue number - <AUTHOR> http://b2brainmaker.com/b2b-lead-generation/the-most-important-revenue-number-you-present-in-a-business-plan/

<comment><USER2> This is BS - if a company knows all the answers to these questions, then the company is not innovating. The hardest thing about innovation is finding out who else believes your vision. The assumption that a good business has answered all these questions is the assumption that this good business is not innovating. Revenue forecasting for new products is hard, and investors who add value to high risk start ups must recognize the risk in innovation. If you want answers to these questions, either insist on bullshit stories or invest in old school businesses. No risk = no return.

<comment><USER3> If you ask a company those questions and the answer you get is "we have no idea, we're innovating", wouldn't you agree that's a flag you ought to investigate? Eliminating risk in business isn't your goal as a leader or investor, but managing risk is.

A fool and his money are soon parted.</comment></comment>


A terrible, horrible, no-good, very bad day at Slack - <AUTHOR> https://slack.engineering/a-terrible-horrible-no-good-very-bad-day-at-slack-dfe05b485f82 <comment><USER2> TL;DR First a performance bug was caught during rollout, and rolled back within a few minutes. However this triggered their auto-scaling of web apps to ramp up to more instances than a hard limit they had. This in turn triggered a bug in how they update the list of hosts in their load balancer, causing it to not get updated with new instances, and eventually go stale. After 8 hours the only real remaining instances in the list were the few oldest ones, so when they then scaled down the number of instances again, these old instances were the first to be shut down, causing the outage, because the instances that should have taken over were not listed in the stale load balancer host list.

<comment><USER3> It almost seems like no one group understands the system as a whole, so when one part fails, no one has a clear idea of the domino effects that can happen. I'm guessing this is the result of really complex systems interactions.

<comment><USER4> Which is what always happens when the company aims to build an "engineering playground" full of microservices and other moving parts without a clear technical justification and balancing the pros & cons and why I personally don't like working on such projects - it makes me feel uneasy not having a good understanding of the entire system.

To be fair to Slack, at their scale, lots of moving parts might make sense, but I see a lot of companies (including startups with very few customers) going down the microservices route and exposing themselves to such a risk when there is no major upside beyond giving engineers lots of toys to play with and slapping the "microservices" and related buzzwords on their careers page.

<comment><USER5> I think you are drawing the wrong conclusions here... Microservices is not the boogyman here. It more likely has to do with speed of development, developer turnover and a plethora of other things that result in insufficient knowledge transfer.

Microservices (like just about anything) can be implemented well or poorly. There's a reason we have sophisticated orchestration solutions like Kubernetes... it exists to tame large scale deployments that has sensible failover processes.

The benefits you get are services that can be scaled independently, deployments that only affect isolated pieces of code, horizontal scaling, dockerized environemnts, etc. All of these advantages should exist in well designed systems, but systems that have been executed hastily will likely have critical problems crop up at some point.

<comment><USER6> I like your statement but I don't think it has to do with the speed of development either.

In a monolithic architecture the devs that deal with it, have to deal with the program as a whole. So if something doesn't work, its their problem. Where as in a micro service architecture it can be easy to spin up a service and not know the systems that integrate with it.

The problem here is with documentation and understanding of architecture. Its just the nature of the beast that the monolithic dev knows how thing communicate with the monolithic program because he needs to know, in order to do his job. In this instance the problem isn't with micro services, its with the execution. And that execution is a very easy trap to fall into with micro services.

<comment><USER7> The same thing can happen in a monolith. Make a change, run tests, and make sure your new feature works, while breaking some untested behavior you didn't know about in some other part of the monolith.

</comment></comment></comment></comment></comment></comment><comment><USER8> "I'm still not understanding why it's so hard to display the birthday date on the settings page. Why can't we get this done this quarter?"

Look, I'm sorry, we've been over this. It's the design of our back-end. First there's this thing called the Bingo service. See, Bingo knows everyone's name-o, so we get the user's ID out of there. And from Bingo, we can call Papaya and MBS (Magic Baby Service) to get that user ID and turn it into a user session token. We can validate those with LNMOP. And then once we have that we can finally pull the users info down from Raccoon.

<comment><USER9> Reference: [https://www.youtube.com/watch?v=y8OnoxKotPQ](https://www.youtube.com/watch?v=y8OnoxKotPQ)

I revisit this video every now and then.

<comment><USER10> This one always cuts too close to home

[https://youtu.be/_o7qjN3KF8U](https://youtu.be/_o7qjN3KF8U)

<comment><USER8> Can't wait for BallmerCon this year. The XLOOKUP meta is going to spice things up.

[https://www.youtube.com/watch?v=xubbVvKbUfY](https://www.youtube.com/watch?v=xubbVvKbUfY)

</comment></comment></comment></comment><comment><USER11> I had trouble getting through this article because my internal monologue was screaming "Envoy and xDS wouldn't have this problem". But that's exactly what they decided ;) HAProxy is a little behind the state of the art on "hey I could just ask some server where the backends are", and it shows in this case. (The "slots" are particularly alarming, as is having to restart when backends come and go.)

xDS lets you give your frontend proxy a complete view of your whole system -- where the other proxies are (region/AZ/machine) and where the backends are, and how many of those. It can then make very good load balancing decisions -- preferring backends in the AZ that the frontend proxy is in, but intelligently spilling over if some other AZ is missing a frontend proxy or has fewer backends. And it emits metrics for each load balancing decision, so you can detect problems or unexpected balancing decisions before it results in an outage. [https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv...](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/zone_aware)

I also like the other features that Envoy has -- it can start distributed traces, it gives every request a unique ID so you can correlate applications and frontend proxy logs, it has a ton of counters/metrics for everything it does, and it can pick apart HTTP to balance requests (rather than TCP connections) between backends. It can also retry failed requests, so that users don't see transient errors (especially during rollouts). And it's retry logic is smart, so that if your requests are failing because a shared backend is down (i.e. your database blew up), it breaks the circuit for a period of time and lets your app potentially recover.

The result is a good experience for end users sending you traffic, and extreme visibility into every incoming request. Mysterious problems become easy to debug just by looking at a dashboard, or perhaps by clicking into your tracing UI in the worst case.

The disadvantage is that it doesn't really support any service discovery other than DNS out of the box. I had to write github.com/jrockway/ekglue to use Kubernetes service discovery to map services to Envoy's "clusters" (upstreams/backends), but I'm glad I did because it works beautifully. Envoy can take advantage of everything that Kubernetes knows about the service, which results in less config to write and a more robust application. (For example, it knows about backends that Kubernetes considers unready -- if all your backends are unready, Envoy will "panic" and try sending them traffic anyway. This can result in less downtime if your readiness check is broken or there's a lot of churn during a big rollout.)

<comment><USER12> Isn't Ambassador doing the same thing?

Btw not sure if you read till the end, they are actually in the process of migrating to Envoy.

<comment><USER13> Ambassador is a Envoy control plane, as in it uses Envoy for doing the data proxying, but it sets it up.

So yes, it is :)

</comment></comment></comment><comment><USER14> Super interesting post. Following blog links, the timeline in [https://slack.engineering/all-hands-on- deck-91d6986c3ee](https://slack.engineering/all-hands-on-deck-91d6986c3ee) also offers a look at the play by play.

However, as far as I can read it, they have somewhat different views on the root cause?

"Soon, it became clear we had stale HAProxy configuration files, as a result of linting errors preventing re-rendering of the configuration."

vs.

"The program which synced the host list generated by consul template with the HAProxy server state had a bug. It always attempted to find a slot for new webapp instances before it freed slots taken up by old webapp instances that were no longer running. This program began to fail and exit early because it was unable to find any empty slots, meaning that the running HAProxy instances weren't getting their state updated. As the day passed and the webapp autoscaling group scaled up and down, the list of backends in the HAProxy state became more and more stale."

Maybe a combination of the two?

<comment><USER15> Honestly it's a bit tough for me to parse, but the way I'm reading it,

1\. Stale configs led to an overabundance of web apps, and _then_

2\. Old instances of the web app couldn't be removed because of the consul- template bug.

so, yes, a combination (in sequence) of the two.

Hard for me to be sure because I'm by no means knowledgeable on this stuff.

<comment><USER16> Even easier way to understand what happened:

\- slots full

\- to update slots with a new host you need an empty slot

\- hosts went away but updating config was impossible -> errors because config referenced non-existing hosts

<comment><USER17> Agree but one more:

\- monitoring was broke so we didn't learn about it until it was too late

</comment></comment></comment></comment><comment><USER18> > " The reason that we haven't been doing any significant work on this HAProxy > stack is that we're moving towards Envoy Proxy "

I truly don't understand the cycle of churn. Once most edge cases and bugs of Haproxy have been found, the right decision is not migrating to completely unknown territory again. No project is a silver bullet, and changing stacks after you find the bugs makes for a terrible return on your bug-hunting investments.

<comment> brown9-2 I think if you read this as they're moving to Envoy _because_ of this incident then you've misread.

But it also sounds like Envoy and HAProxy have fundamentally different approaches to service discovery:

[https://twitter.com/mattklein123/status/1277729102271676416?...](https://twitter.com/mattklein123/status/1277729102271676416?s=21)

[https://twitter.com/mattklein123/status/1278114953497436161?...](https://twitter.com/mattklein123/status/1278114953497436161?s=21)

</comment></comment><comment><USER19> Great writeup. It's cool that they were able to figure it out as quickly as they did, all things considered.

If I were brought in as a consultant on this, my first question would be: why are you using a fleet of HAProxies instead of the ALB? I'm not saying that's a bad choice, but I'd want to know why that choice was made.

The second question I would ask is what kind of Chaos Engineering they are doing. Are they doing major traffic failover tests? Rapid upscaling tests? Random terminations?

Those are probably the first two things I'd want to solve.

<comment><USER20> I hope all of these questions would be asked only after everything was working again!

<comment><USER19> Of course. :) These are things I would ask during a post-mortem.

</comment></comment></comment><comment><USER21> Interesting bit:

> The program which synced the host list generated by consul template with the > HAProxy server state had a bug. It always attempted to find a slot for new > webapp instances before it freed slots taken up by old webapp instances that > were no longer running. This program began to fail and exit early because it > was unable to find any empty slots, meaning that the running HAProxy > instances weren't getting their state updated

</comment><comment><USER22> I've just been bitten by this too:

      The broken monitoring hadn't been noticed partly because this system 'just worked' for a long time, and didn't require any change.

Any experience on how to deal with it? Who watches the watchers?

<comment><USER12> Chaos Engineering comes into play here. Deliberately break your platform to see if everything works as expected.

<comment><USER23> But how, exactly?

Sure, the chaos monkey could kill haproxy-server-state-management but that wouldn't uncover the bug in question — it'd just demonstrate that without it running HA proxy's view of the world goes stale, which anyone would expect. Triggering the bug would require reducing the number of HA proxy slots below the number of webapps running for many hours. This is clearly something chaos engineering _could_ do but IMO it's highly unlikely anyone would think to do this. If they thought of this they would also have thought about adding tests that caught such as issue long before the code went into production.

In my experience chaos engineering is often only as good as the amount of thought put into the things it does. Killing processes here and there _can_ be useful but it often won't expose the kind of when-the-stars-align issues that take down infrastructures.

It looks like a classic lack of monitoring, as the article says. Alerting on webapps > slots, early exits, or differing views of the number of webapps up would have likely caught this.

<comment><USER12> > Sure, the chaos monkey could kill haproxy-server-state-management but that > wouldn't uncover the bug in question

No it won't. But it would uncover their missing alerts for a critical platform component. Their issue was exacerbated by the fact that state-management kept failing for nearly 12 hours and no one noticed.

<comment><USER23> Maybe, though again I would expect an alert for the process being dead would also not uncover this particular issue. It's possible someone would notice that there was insufficient alerting then go back to add something which would have caught this but it's far from certain. OTOH it's also possible that at this stage in the game, when the code and systems are EOL, that something like chaos testing is the last chance to catch this problem.

I'm not totally against chaos testing. I just haven't seen it done well and think it's actually pretty hard to pull off (particularly the non-technical aspect of convincing people it's okay to let this thing go mad). I'd love to see how effective it was within Netflix.

</comment></comment></comment></comment></comment><comment><USER24> > It's worth noting that HAProxy can integrate with Consul's DNS interface, > but this adds lag due to the DNS TTL, it limits the ability to use Consul > tags, and managing very large DNS responses often seems to lead to hitting > painful edge-cases and bugs.

I was surprised how they dismissed HAProxy integration with Consul using SRV DNS records. Can anyone confirm the problems they highlight?

It seems like their service that broke would not be needed if they went the DNS route.

<comment><USER25> Pre 2.0 there were a few bugs with SRV discovery, maybe they adopted early and got bit? Just an anecdote but we've been using it since 1.9 without issue. Massively different scales though.

Pre k8s and before srv support we used consul template in prod as well but it always scared me, seemed like too many moving pieces for what should've been a simple system.

<comment><USER25> I asked internally and figured out the gotcha that bit us: default dns payload size is 512b, which is enough for a few backend hosts but for sure not 12 or 30. Limit is 8kb, which probably wouldn't work for whatever slack is doing.

[https://cbonte.github.io/haproxy- dconv/2.1/configuration.htm...](https://cbonte.github.io/haproxy- dconv/2.1/configuration.html#5.3.2-accepted_payload_size).

Because DNS records come back in random order for each response, those truncated dns responses caused the backend slots to constantly rotate between different pod instances. Haproxy was graceful about the rotations, but it showed up as suddenly very strange latency / perf numbers when a backend was scaled up to say 10 instances from the normal 3

</comment></comment></comment><comment><USER26> Seems to me that most of the problems came from the sheer scale Slack operates at. A single instance of a self-hosted chat application wouldn't require any of the load balancing infrastructure. But SaaS is really convenient. I wonder if there's a market for "local cloud" companies that operate out of your city and offer hosting of popular open source projects. The complexity would be much lower, and hopefully the reliability higher. Plus you get the benefit of lower latency. 90% of the people in my team's slack channels live with 5 miles of each other.

<comment><USER27> A single instance of a self-hosted chat application would also result in unbounded hours investigating some obscure bug that affects 1 in 100,000 deployments, etc; while the developers don't know because they don't have enough ability to remotely dig in and investigate.

Self hosting is not a pancea and I would not think it would be more reliable.

There's also limited reason why you really want your servers in your city, 5ms of latency savings isn't it,instead of economies of scale in large datacentres with good network uplinks and centralized reliability teams.

</comment></comment><comment><USER28> Autoscaling is hard. Never ever use one that you don't thoroughly understand.

An autoscaler that keeps chugging when deploys aren't green is outright dangerous.

<comment><USER12> I think this one is more of a service discovery bug than auto-scaling.

<comment><USER28> Perhaps you know more than is in this blog post, but it sounds more like a rather standard load balancer.

Deployment broke. Yeah, that probably should have been caught. Even if it didn't, monitoring should have caught the stale load balancer config. It didn't, for some reason unknown to us.

These things happen. Things break. The autoscaler then proceeded to kill customer traffic. That was the part that worked as designed, so another design would have avoided escalating the situation (if you forgive some armchair engineering here).

<comment><USER29> Maybe what's missing was to test high load and autoscaling to 5x the traffic on, say, a holiday (when Slacks customers don't work)

</comment></comment></comment></comment><comment><USER30> > One of the incident's effects was a significant scale-up of our main webapp > tier.

Sorry, I'm not very familiar with the terminology here; what is the "main webapp tier"?

<comment><USER31> Apps these days might have several groups of services. For a simple case you might have a web tier serving http requests from customers, and you might have a Worker/background task tier. They usually scale independently.

</comment></comment><comment><USER32> Another insight for me was coding the original system to have N slots. This should've been a red flag—why is there an arbitrary constraint there? Why allow the system to have a fixed limit there?

If you choose to go down the slots road, then you need to put in alerts and discovery for reaching slot limits—-which means monitoring and tracking them, then setting up alerts.

</comment><comment><USER33> What's the differences of using HAProxy or Envoy between using the cloud load balancers of AWS or Google Cloud?

<comment><USER34> Cloud load balancers can be sneakily expensive. Few months ago, we spent a few weeks replacing an ELB with naive client side load balancing via round robin, which saves us > 200k/year. ELBs charge per byte transmitted, which seems reasonable, but can end up really expensive.

<comment><USER29> > client side load balancing

In the browser? Or a mobile app?

They send 1 api req to server 1, then 1 to server 2 and so on? What about any session cookies maybe tied to a specific server?

<comment><USER4> Presumably round-robin DNS. A DNS response would only return a handful of servers, of which the client will itself only pick one at random for the duration of the session.

Now this approach has drawbacks (DNS responses are cached, and the DNS record picked initially by the client will typically be cached until the app/browser is restarted) but if they are acceptable to you then it's an easy, proven solution.

<comment><USER29> Hmm I'd guess they have DNS cnames like api1.x.com and api2 and 3, 4

And then the client picks one, and if that server is offline, picks another

Seems as simple as DNS based? And works with broken server(s)

<comment><USER35> Except you don't need that because you can just return all four IP addresses for one record, e.g. api.x.com

<comment><USER29> I think if such a DNS/ip based round robin server is down, or replies 500 error, the client won't try another server

Unless there's a way to get all ip addrs in js? By custom client code that queries the DNS system?

<comment><USER4> DNS resolution is handled by the DNS server and the browser. JS isn't involved, it's just telling the browser to connect to a certain hostname and the browser itself decides which IP to map it to (based on its DNS cache).

If the DNS server is down the website wouldn't load at all, but this is an acceptable trade-off considering DNS is a very simple system (not many things can go wrong) and servers can be redundant.

<comment><USER29> There's a misunderstanding. That reply is to me off topic, I knew about those DNS things already

Thanks anyway for replying

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER36> Stupid question, and admittedly off topic:

What's with the "terrible, horrible, no-good, very bad" expression I see a lot? It's a reference to something? From googling, it seems to be this [1], but ... _why_? Why do people reference it?

Usually you reference some work like this because a) the phrase is unusually creative, or b) the work is unusually memorable. Neither is true here.

[1] [https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Ho...](https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Horrible,_No_Good,_Very_Bad_Day)

<comment><USER18> This book sold well enough to score a tv series, a disney movie, a musical, and a theater play.

How much more successful do you need it to be? Odds seem very high that a kid growing up in the past 50 years was exposed to this story and phrase

<comment><USER36> I wasn't. Or if I was, it obviously didn't make an impression, and properly so: combining random negative words isn't creative.

<comment><USER37> It seems there's some kind of "all your base are belong to us" weird resonance for it. I had my own explanation why it works, but the only proof that's really needed is that it became a meme.

</comment></comment></comment></comment><comment><USER38> Pretty fantastic case study in the perils of complex systems. Is there any place where these types of post-mortems are collected? Could be a very valuable resource for systems engineers.

<comment><USER39> Here's a decent list: [https://github.com/danluu/post- mortems](https://github.com/danluu/post-mortems)

</comment></comment><comment><USER40> Scalability is always such a spur of the moment implementation at a startup. This seems to be cruft left over from that startup phase. Would a scalability audit have caught it? Tough to say as Slack came from that build fast and break things era.

<comment><USER29> Scale fast and break things

</comment></comment><comment><USER41> Discussion during the outage: [https://news.ycombinator.com/item?id=23161623](https://news.ycombinator.com/item?id=23161623)

</comment><comment><USER42> I bet if this story was told as an animation on YouTube it would be popular.

</comment><comment><USER43> The title is an allusion to a popular children's book[1]. I'm assuming that an automated algorithm pulled the "very;" hopefully the mods will consider restoring it.

[1] [https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Ho...](https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Horrible,_No_Good,_Very_Bad_Day)

<comment><USER44> HN automatically mangles titles in various ways I'm not fond of, dropping words that might or might not be significant, fiddling with capitalisation, _& c._, but the submitter can go back and edit the title back to what it was supposed to be, and the one time I've done that the system didn't mangulate it again.

<comment><USER45> That's by design, because the software is obviously imperfect. It does more good than harm, though, so we keep it.

</comment></comment></comment><comment><USER46> This is one of the biggest arguments I see for serverless (AWS Lambda + DynamoDB) or at least managed PaaS systems (Google App Engine, Heroku with RDS or CloudSQL). These systems may seem to cost more for some workload curves (or might even be cheaper for your curve), but the difference is worth it because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and by definition they're already familiar with running workloads orders of magnitude bigger than yours. Even then the platforms might be cheaper because you're only paying for a fraction of the dev-ops team's salaries, but you get their full benefit.

<comment><USER47> > because you're paying for specialized 24/7 dev-ops teams whose only job is > to keep these systems running smoothly, and by definition they're already > familiar with running workloads orders of magnitude bigger than yours

This is based on faith — there might, or might not be a specialized 24/7 devops team who runs these things better than you.

My rational mind has trouble accepting things based on faith, which is also why I don't trust RDS: I don't know of any way to run a distributed SQL database without data loss (neither does Jepsen), so why would I expect RDS to do this correctly?

Using those services does provide a warm and fuzzy feeling, though.

<comment><USER48> The specialized 24/7 devops team (if it's there) also has a few thousand other customers instead of being there just for you. They might have other priorities at any given moment. It's not like AWS or GCP are renowned for the quality of their customer service.

<comment><USER46> This is actually a plus point if you ask me. These few thousand customers are all operating on the same racks as me - in an environment like AWS or GCP there's no special part of the datacenter reserved for fancy customers. The billion-dollar customers all run their VMs on the same rack as me. So whatever work the ops teams do to keep things reliable benefits me as much as the biggest customers.

<comment><USER49> Slack's problem though had nothing to do with racks going down.

I'm not convinced Amazon's team is immune from the sort of complex failure mode described here. 'll bet there's people with equivalent sorts of stories about where edge cases in service interactions (either their own set of Lambda services or the AWS ones behind them, or more likely both) lead to a similar unexpected failure cascade.

<comment><USER50> > I'm not convinced Amazon's team is immune from the sort of complex failure > mode

You're being way too kind. Not only is AWS not immune, their autoscalers are often absurdly primitive. Like, hourly cron job doubling / halving within narrow safety rails primitive, where it's not merely possible to find a load that trips it up, it's all but inevitable.

This varies by service, but they always project an image of their infrastructure being rather smart, and in the cases where I've been able to make an informed guess about what's actually going on, it's usually wildly inconsistent with the marketing. They don't warn you about the stinkers and even on services with good autoscaling and no true incompatibility between AWS's hidden choices and your needs, your scaling journey will involve periodic downtime as you trip over hidden built in limits and have to beg support to raise them. Sometimes you get curiously high resistance to this, leading to the impression that these aren't so much "safety rails" as hardcoded choices.

Oh, and just last week we managed to completely wedge a service. The combination of a low limit on in-flight processes, two hung processes, immutability on running processes, and delete functionality being predicated on proper termination led to a situation where an AWS service became completely unusable for days while we begged support to log in and clear the hung process. Naturally, this isn't going to count as downtime on any reliability charts, even though it's a known problem and definitely looked a lot like downtime on our end.

We're a small (<10) team with modest needs. AWS lets us do some crazy awesome things, but it really bugs me how reliably they over-promise and under- deliver.

<comment><USER49> > You're being way too kind. Not only is AWS not immune, their autoscalers are > often absurdly primitive.

Yep. Very much so. Mostly because I don't have enough personal Lambda-specific warstories to feel confident badmouthing it in the context of this discussion thread. But the bits of AWS I do use are certainly not all rainbows and roses...

I have one app/platform I run that basically sits at a few requests an hour for 11 months of the year, then ramps up to well over 100,000 requests a minute between 8am and 11pm for 14 days. Classic ELB (back in the day) needed quite a lot of preemptive poking and fake generated traffic to be able to ramp up capacity fast enough for the beginning of each day (aELB is somewhat better but still needs juggling). We never even got close to getting autoscaling working nicely on the web and app server plane to let it loose in prod with real credit card billing at risk, we just add significantly over provisioned spot instances for our best estimates of yearly growth (and app behaviour changes) for the two weeks instead, and cautiously babysit things for the duration.

It's nice we can do that. It'd be nicer if I didn't have to keep explaining to suits and C*Os why they can't boast own the golf course that they have an autoscaling backend...

<comment><USER50> Hey, let's not be hasty, it sounds like you've built a bespoke autoscaling backend that intelligently predicts future usage and dynamically allocates compute resources to match customer needs.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER51> Why was the "very" removed? It sounds so much better like that.

<comment><USER45> HN's debaiting software took it out. We put it back.

[https://news.ycombinator.com/item?id=23756384](https://news.ycombinator.com/item?id=23756384)

[https://quoteinvestigator.com/2012/08/29/substitute- damn/](https://quoteinvestigator.com/2012/08/29/substitute-damn/)

</comment></comment><comment><USER52> After I used Discord in different contexts for months now (and Slack for years), I can't understand why someone willingly chooses Slack.

It's the Atlassian of chat tools. Horrible performance and bad usabillity.

<comment><USER53> I use Zulip for the day to day (it's amazing, I can't recommend it enough), but sometimes use Slack because some open source communities use it, and I'm always amazed at how damn slow it is. I can consistently out-type it, it's terrible.

I guess it was great when it started out, but they're slowly boiling the frog, who is us.

<comment><USER54> Zulip won my last bake-off for chat systems. Integration was easy and the topic method of providing threads was amazing. The only feature it was missing was federation. In the XMPP world, you could communicate with users on other XMPP instances. With Zulip, you can only communicate with local users. Do you know if this is still the case?

<comment><USER55> As other folks have mentioned, Zulip has a number of cross-server integrations with both the Zulip protocol and other protocols like XMPP. There's a few we document here as well as Matterbridge:

* [https://zulipchat.com/integrations/communication](https://zulipchat.com/integrations/communication) * [https://github.com/42wim/matterbridge](https://github.com/42wim/matterbridge)

We'll eventually add a more native way of connecting a stream between two Zulip servers; we just want to be sure we do that right; federation done sloppily is asking for a lot of spam/abuse problems down the line.

(I'm the Zulip lead developer)

</comment></comment></comment></comment><comment><USER56> Not being involved in this kind of scaled-up devops kind of stuff, my read of this article was "HAProxy sounds _awful_ "... but they noted in the article

1) Migrating to better software, and 2) Newer versions of HAProxy are _also_ better software.

But yeah, the first half where they're discussing "we don't do X, Y, or Z clean, idempotent way of doing config because the HAProxy doesn't perform right if you do it that way." sounded painfully familiar for a bunch of awful tools that I've used.

<comment><USER57> I don't think that's fair. HAProxy is good and reliable. It's just old for the use cases we need today. Even the article has this opinion:

> While HAProxy has served us well and reliably for many years, it also has > some operational sharp edges

Saying it sounds awful is like saying Vanilla JS sounds awful because it doesn't scale to the demanding webapp use cases of today.

And, as you mentioned, it's really compounded by the fact that they're using an outdated version of HAProxy, but that doesn't make it awful software.

<comment><USER56> > Vanilla JS sounds awful

... it kinda is?

<comment><USER57> I don't think so. Just because something isn't a swiss army knife doesn't make it awful in my book.</comment></comment></comment></comment>


Comcast introduces Xfinity Mobile - <AUTHOR> http://corporate.comcast.com/news-information/news-feed/comcast-xfinity-mobile <comment><USER2> God, now they're going to require me to have one of their mobile plans along with cable, home phone, and internet just to watch sportsball... Christ.

</comment><comment><AUTHOR> Link to the video they sent out with the email promotions [https://static.cdn.responsys.net/i5/responsysimages/content/...](https://static.cdn.responsys.net/i5/responsysimages/content/comcast/xfinitymobileV.mp4)</comment>