Show HN: Sundown – an alternative to Markdown - <AUTHOR> https://seyedi.org/blog/sundown <comment><USER2> I'm iffy on this kind of thing. Markdown has the advantage of looking roughly like what you'd do in plain text and is completely readable in plain text with no processing. It also has the advantage of being nearly standard; it is used on thousands of sites, including some very large ones (e.g. reddit, github, etc.). If I wanted my users to memorize HTML tags I would have just gone with HTML.

This seems to have the negatives of both Markdown _and_ HTML, and no unique advantages over either. It's a little less verbose than HTML, but you have to know HTML tags (and Sundown syntax) to use it, and I have my doubts it can avoid the quirks Markdown has because converting any "friendly" markup to a formal syntax like HTML is always prone to quirks. It has the alleged advantage over Markdown of being able to use HTML and Sundown together...but, some Markdown processors also allow mixing MD and HTML, so this is not a unique benefit (this is quirky in Markdown, but I suspect it's quirky in Sundown, too).

Maybe I'm missing the benefit here, but it seems like a technical solution to a people problem, and that almost never works.

Also, as an aside, it's got the same name as a Markdown processor that is even mentioned in the article! That seems bound to cause confusion, given the similarities in purpose.

<comment><AUTHOR> Author here.

> Maybe I'm missing the benefit here

Sundown is useful because it lets you abstract your custom server-side HTML tags. In other words, you can extend the language very easily.

For example, you can create a tag called "note" and use that instead of repeating HTLM, CSS and even JS:

        <s:tag name="note">

                Note: {{@content}}!!

        </s:tag>

And then call your new tag using the HTML syntax or Sundown:

        <note>Sundown is awesome ;)</note>
        :note Sundown is awesome :)

Result:

Note: Sundown is awesome :)!!

Sundown is just an HTML preprocessor (like Markdown) once it is transformed to HTML it's the SunSed language that compiles all the custom HTML tags to executable format.

<comment><USER3> I can see the use case you're going for. As others have mentioned though, this value is not coming as an alternative to markdown but rather an enhancement to HTML in a different use case.

Feels kind of like Sundown:HTML::SCSS:CSS. If you can communicate this (vs being an improvement on markdown), you'll probably get people to take a closer look.

<comment><AUTHOR> You are right. Thank you for the insight! How about this:

Sundown and Markdown are both HTML preprocessors with different goals.

<comment><USER4> Not all markdown processors implement inline HTML, which I learned to my detriment recently (the "markdown" package on npm doesn't).

<comment><AUTHOR> Exactly. Most Markdown processors don't implement this feature because it is very unpredictable and has too many contradictory edge cases.

<comment><USER5> ... which makes Markdown even _less_ like your software. Also less relevant to marketing efforts as people up-thread are saying. Some of the use-cases that you're covering with your software were never covered by Markdown, nor were they ever intended to be use-cases for Markdown. So you're not going to find people that are currently using Markdown that will need Sundown for those use- cases.

</comment></comment></comment></comment></comment></comment></comment><comment> git-pull First off, Good work on your project, Sed!

Here are some clickables of SunSed:

\- [https://www.sunsed.com/](https://www.sunsed.com/)

\- [https://www.sunsed.com/pricing](https://www.sunsed.com/pricing)

\- [https://www.sunsed.com/tags/s](https://www.sunsed.com/tags/s) A list of builtin tags for Sundown

I am fine with a new markup if it's quick and to the point. I believe that we need to give the markdown/reStructuredText things a fresh look. At the very least, we may come up with ideas to improve existing languages.

Are you planning on opening the source to Sundown? What's the current implementation running in? (golang, python, c, etc?)

Some of the other commenters are mentioning markdown being fast. This is probably true. Since there are processors for markdown in C. Actually, when I first glanced the title, I thought it was [https://github.com/vmg/sundown](https://github.com/vmg/sundown).

In any event, that's just a naming issue. The bigger point is, I think the overall project at SunSed looks pretty cool. Simple markup turns into a nice website. Congratulations on having the website up!

<comment><AUTHOR> Sed here.

Thank you very much!

> Are you planning on opening the source to Sundown?

Sundown is just a HTML preprocessor implemented in a few lines of regular expressions! I actually planned to open source it before writing the article but did not have the time.

However, I'm not planning on open sourcing the SunSed language compiler (the part that allows you to create custom html tags). But I will open source all the tags and libraries here:

[https://github.com/sunsed](https://github.com/sunsed)

</comment></comment><comment><USER6> Markdown is being critized because it doesn't allow markdown syntax within HTML blocks, and because the HTML vocabulary recognized isn't customizable. Both problems were basically solved by SGML over 30 years ago.

In SGML, you can define arbitrary tokens (character sequences) that the parser replaces into something else, typically into start-element or end-element tags. Replacement text rules are defined in a context-sensitive way depending on the parent element wherein token(s) to replace appear as child content. For example, an asterisk could be mapped to , and within an  element could be mapped to , terminating it, and thereby implementing a small fragment of markdown.

This, together with tag omission/inference rules on a fully custom DTD vocabulary powerful to cover HTML5, gives you quite a formidable technique for markdown customization, based on an ISO standard.

</comment><comment><USER7> Very poor naming choice. Sundown is the name of a popular Markdown implementation:

[https://github.com/vmg/sundown](https://github.com/vmg/sundown)

Strongly suggest renaming your project as early as possible.

<comment><AUTHOR> Author here.

You are right about the name confusion! It has been several years since the last commit to the /vmg/sundown and even if it was active it's not a very common Markdown implementation (although it's one of the best).

<comment><USER8> How often does one need to commit to a project in order to not have it's name stolen?

<comment><USER9> Please don't post acerbic swipes to Hacker News. Even if you have a point, it's not worth damaging the community for.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

[https://news.ycombinator.com/showhn.html](https://news.ycombinator.com/showhn.html)

</comment></comment></comment></comment><comment><USER10> Here's the author's intent with their system:

> ... create a platform that makes it super easy for developers to create > awesome websites without forcing them to learn new stuff (language, syntax, > platform, framework and more)

And yet:

> I explain what was wrong with Markdown and why I finally had to create my > own Markdown alternative

I do not think a new, custom HTML template language solves the original goal.

<comment><AUTHOR> Sed here (the author).

SunSed is for developers with HTML knowledge. You can use SunSed language to create custom HTML tags (abstract HTML, CSS and JavaScript) and then Sundown to abstract your HTML.

I think Sundown is very easy to learn and SunSed language has very little learning curve because it follows the exact syntax of HTML:

[https://www.sunsed.com/tags/s](https://www.sunsed.com/tags/s)

<comment><USER8> Why not use Pug ?

<comment><AUTHOR> I already answered this. Let me copy and paste it from my other comment:

Aside from the difference in syntax (which is huge), I would say convenience.

SunSed = Hosting + IDE + CDN + a new programming language + Layout and Template system (like Jekyll and Hugo) + Many built-in tags to abstract things like embedding Youtube videos in a responsive way (which is not easy -- requires so many div wrappers).

You can also create your own tags and share it with others. I have started the process of creating tags for useful and popular libraries:

[https://github.com/sunsed](https://github.com/sunsed)

If you know HTML, you already know the syntax of SunSed programming language, and can create fast dynamic websites with ease by abstracting your website content in custom HTML tags!

After all, SunSed is a programmable website builder.

<comment><USER8> I don't understand how all this relates to Pug at all. The word Pug is not even in your comment.

</comment></comment></comment></comment></comment><comment><USER11> I think there's lots of opportunity for a markdown alternative, but I worry that this starts in the wrong place. It seems like it's built around code, whereas Markdown is built around writing.

A friend of mine who works at a newspaper noted that it was really hard to get a lot of people in the mindset of Markdown, because it requires users to think about their writing in terms of HTML, which is harder than it sounds for some. This seems like it pushes things further in a direction where people who aren't code-savvy can't get on board.

The other thing to keep in mind: The reason why Markdown is great is because it's readable even without being rendered. This is not, because it simplifies the HTML without abstracting it. That's a tough hurdle to get past for a lot of folks.

<comment><USER12> I would be much more interested in your friend's view of an ideal text format.

<comment><USER11> I don't think his point was that Markdown even sucks. I think, rather, it's that a lot of people don't know HTML and find rich text much more intuitive, despite its weaknesses. It's sort of the 90/10 rule; Word is good for 90 percent of people, but the other 10 percent are driven nuts by it.

</comment></comment></comment><comment><USER13> Great solution. Given the problem statements that lead to this, i was pleasantly surprised by the elegance of the syntax.

There are the obvious arguments about plain-text readability and additional standards, I would suggest thinking about this not as a Markdown replacement but rather a different use-case entirely. I understand the author framed it as a replacement. While that may have been the impetus to the project, i think the result is something entirely it's own.

Markdown has a place when a document stands to be published/shared in multiple formats. I use it regularly for blog posts that i also email to others for feedback in plaintext. Sundown looks to me like a light-weight, text-focused haml. Usable when something is intended to only be published in HTML, but where you want to be freed from that markup.

<comment><AUTHOR> Thank you!

> I would suggest thinking about this not as a Markdown replacement but rather > a different use-case entirely.

You are right! I'm thinking about the same thing here! I could even give it no name and instead present it as part of the SunSed programming language (the part that lets you create custom HTML tags).

</comment></comment><comment><USER14> The post just seems like a verbose post hoc justification for wanting to reinvent something. Which is fine, a lot of good learning comes out these toy domain languages. I just don't see any technical benefits or conceptual leaps here other than to say "I built this".

It's like back in the day when someone would proclaim smarty to be a disaster and then release their template system that was conceptually exactly the same baring some syntactic sugar

<comment><AUTHOR> > A lot of good learning comes out these toy domain languages.

I agree with the learning part but SunSed is not a toy language. I could be wrong about how valuable it's going to be but it's definitely more than a toy.

Sundown is a very simple preprocessor that plays well with the SunSed language. With SunSed, you can create your own HTML tags (that run server- side) and abstract HTML, CSS and JavaScript.

Sundown is just the icing on the cake that lets you easily use your custom tags. SunSed language is a complete programming language, you can checkout the tags here:

[https://www.sunsed.com/tags/s](https://www.sunsed.com/tags/s)

Again, I could be wrong on how valuable it's going to be for front-end developers. But I strongly feel that it's a little bit more than a toy! ;)

</comment></comment><comment><USER15> [https://www.sunsed.com/policies/until-the- end](https://www.sunsed.com/policies/until-the-end)

> Our promise to you ul.[sic] We are dedicated to supporting our products > forever (or until the last customer turns off the lights). ul.[sic] If we > create a new version of SunSed that breaks backward compatibility we will > support the older version forever. p.[sic]

</comment><comment><USER16> Started reading with interest, but then everything started to look really similar to HAML [0] after a bit. With the exception that HAML also lets you add CSS classes to your tags...

[0] - [http://haml.info/](http://haml.info/)

<comment><AUTHOR> With SunSed you can create your own custom HTML tags and abstract all the CSS and JS then use Sundown to call that tag (or the HTML syntax).

        <s:tag name="note">

                Note: {{@content}}!!

        </s:tag>

And then call your new tag using the HTML syntax or Sundown:

        <note>Sundown is awesome ;)</note>
        :note Sundown is awesome :)

<comment><USER17> [https://www.webcomponents.org/introduction](https://www.webcomponents.org/introduction)

</comment></comment></comment><comment><USER18> As a product marketer, I have to say that the name is very unfortunate.

1\. Sundown is the name of a Markdown library written in C: [https://github.com/vmg/sundown](https://github.com/vmg/sundown). _That_ Sundown is pretty widely known and used among Markdown implementers. As such, when/if the OP's project takes off, it is going to be confusing (for awhile).

2\. Also, the name itself doesn't strongly suggest that it's an alternative to Markdown. Do recall that Markdown's etymology comes from "(HTML) markup". For the Markdown-implemented-in-C Sundown, the name worked because it evoked its relationship with Markdown. In this case, that's not what the project is about. If the intention was to evoke Markdown, then they should have kept the "Mark" part of it, not the "down" part of it.

<comment><USER19> Give people some credit. The name is not confusing. It's called SunDown because it's part of the author's larger project, SunSed.

<comment><USER20> The author himself is inviting the comparison, both in the submission title and on his site.

</comment></comment></comment><comment><USER21> This looks very promising. While I do not see this as a replacement for Markdown this has some potential for use cases where Markdown is less suitable, like messaging.

The explicit use of the colon makes things a bit saner for users - we've all written some 'unexpected' markdown before; between the many different flavors (especially line breaking behavior), numerous symbols, and numerous ways to do the same thing (like underlining, bullet points and images). I can see this being a big relief for end-users for certain use-cases.

I'd love to try it out if I could see some source code (or a live demo I don't have to sign up for). Do you plan on setting up a specification?

</comment><comment><USER22> > I feel like Jeff Atwood and his friends abandoned the Commonmark project > that tried to create a universal specification of Markdown. Even if they > did, I don't think that Github, Stackoverflow, Reddit and other major > Markdown users would ever convert.

The CommonMark project has not been abandoned; the CommonMark C parser was implemented on GitHub earlier this year and is used to render Markdown across all repositories: [https://githubengineering.com/a-formal-spec-for-github- markd...](https://githubengineering.com/a-formal-spec-for-github-markdown/)

<comment><USER23> Came here to say that. Also Discourse just moved to a CommonMark compliant markdown parser this release, so between GitHub and Discourse the user base is already huge.

</comment></comment><comment><USER24> I really like it. I was just thinking of writing a markdown to custom HTML parser for my own needs and found markdown syntax to be lacking to express my HTML tags. I can choose SunDown format for it.

Do you have a parser that converts SunDown to HTML already? I can build it as my first project in Rust while attempting to learn it.

<comment><USER12> > for my own needs and found markdown syntax to be lacking to express my HTML > tags

This is a little strange tbh. The majority of markdown parsers will allow inline html elements too.

If you're trying to do a large structured document, which will need block specific styling, try Pug. (Formerly Jade.) HAML & Slim also solve similar problems and have the benefit of a pretty significant userbase. (All support embedded Markdown)

Sundown is looking very much like a naive and poorly thought out also ran.

<comment><USER24> I'm admittedly naive in this front of anything JavaScript/npm/gulp/yarn/many other tools. They always seem to be over complicated for my interest.

Quickly reading about pug feels like it converts templates to some JS files which I don't want.

I'd like a Markdown/SunDown/custom markup to one inlined HTML, which I can deploy wherever I want..

<comment><USER12> Haml or Slim are a better bet than Pug then.

Nb: pug aka Jade used to convert direct to HTML, I also lament the "everything the JavaScript" silliness.

</comment></comment></comment></comment><comment><USER25> I don't think this feels like it's solving any problems that Markdown isn't already solving. If I'm reading it right, I have to do all _this_ just for some italics?

: There is some italic text in this line.

This is seven more characters than the equivalent Markdown! And two more than the equivalent HTML. And includes at _least_ five characters that are a bitch to type on most screen keyboards.

<comment><AUTHOR> Sed here, (the author)

Image tag in Sundown is easier than Markdown:

        :s:img http://link-to-image.jpg  image alt tag

Or how about embedding a tweet:

        :twitter:embed  http://link-to-tweet

Or, youtube, user sign-up form and more...

You are absolutely right! There is a trade-off. Some tags are easier in Markdown (e.g. bold, italic, inline code and a few others). But you cannot extend markdown with your own custom elements can you?

Sundown by itself is not that useful. It gets really useful when you use it as a preprocessor for your own custom HTML tags.

Here I explain how you can create your own server-side HTML tags:

[https://www.sunsed.com/manual/creating- tag](https://www.sunsed.com/manual/creating-tag)

Again, Sundown is just a HTML preprocessor you can write all of the above tags like this:

        <s:youtube>
            http://youtube-link
        </s:youtube>

<comment><USER26> This seems to be aiming for completely different goals though.

People use markdown as a text and word formatting system that is readable in plain text and easily transforms into HTML.

The the main point you are driving to people in this thread are the custom tags which goes against the plain text readability and adds extra complexity in this scope. You are competing against HTML preprocessors that are primarily used for that reason - to be converted into fully fledged HTML. Markdown is not in this scope by any means.

</comment></comment></comment><comment><USER27> This is fine, but isn't really an alternative to markdown. It's not good for what markdown is good for, which is writing and reading plain text that can also be transformed to HTML.

It looks like it's an alternate syntax for HTML angle-bracket tags. (Personally, I see no value in that. I can see it's arguably "cleaner"-looking, but not by much and at the substantial cost of a non- standard syntax.)

</comment><comment><USER28> It looks like he set out to solve all the problems reStructuredText solved a long long time ago

</comment><comment><USER29> I'm not sure the world needs a new markup syntax, especially an awkward mix of HTML and Markdown. But I could be missing the point, since I just base this on my own needs.

But thing is, talking about performance and scale goes exactly against your project's credibility. These are pretty much the two dumbest criterias when it comes to markup syntaxes.

</comment><comment><USER30> FYI: I've started work on an alternative to markdown called texti (text with instructions). It's really more of an evolution e.g. some deletions, some additions, some changes. See some examples at the project site - [https://texti.github.io](https://texti.github.io)

<comment><USER11> Some feedback: As someone who types 30 articles in Markdown per week, # is the deal breaker for me here. It's at a spot on the keyboard that is easy to type. The = sign is not. You should consider a symbol on the left side of the keyboard that is easy to reach. The equals sign just doesn't encourage the kind of muscle memory that the hashtag does.

There are some good ideas here, though. I would like to suggest that there has to be a good way to implement class and id tags in a language like this. That is the only real issue I have with the Markdown spec.

<comment><USER30> Thanks for your feedback. Greatly appreciated. The reason for `=` is to welcome the Wikipedia writer crowd and `#` in the unix world is always used for comments (and "classic" markdown has no comments). What's a human writer- friendly markup convention without comments? Anyways, thanks again for your comments.

</comment></comment></comment><comment><USER31> Seems closer to Pug with :colons than an alternative to markdown to me. Looks like an interesting project though!

</comment><comment><USER32> I really like this. Might use this for my blog as HTML disrupted the flow of typing and yet I couldn't find a satisfactory way to do it with Markdown.

</comment><comment><USER33> Just because _someone_ is going to link it, I might as well do it first: [https://xkcd.com/927/](https://xkcd.com/927/)

But yeah, "Too many implementations of Markdown" is not an issue a new markup language is going to solve.

"Markdown is slow!" is nonsensical: its implementations are slow (it seems the author tested this, but the link supposed to supply evidence doesn't -- I find it hard to believe that aren't fast implementations of Markdown, but _perhaps_ ). Nothing in the language makes it inherently slow.

<comment><USER34> The reference implementation of CommonMark, cmark ([https://github.com/commonmark/cmark](https://github.com/commonmark/cmark))

> cmark can render a Markdown version of War and Peace in the blink of an eye > (127 milliseconds on a ten year old laptop, vs. 100-400 milliseconds for an > eye blink). In our benchmarks, cmark is 10,000 times faster than the > original Markdown.pl, and on par with the very fastest available Markdown > processors.

</comment></comment><comment><USER35> The HTML part of this spec is strange. Most of the practical point of markdown has been to allow a limited set of user-controlled formatting without requiring an HTML parser in the browser. Furthermore, HTML has non-significant whitespace whereas markdown does, so mixing the two will give weird results.

</comment><comment><USER36> I like sundown a lot and see how it is better then markdown. It is very bold and smart reimagining of plain text enriched by tags.

It has a long road ahead, most likely will fizzle but that doesn't mean we can't use it.

</comment><comment><USER37> This doesn't seem to solve any issues that markdown has and also fails to see how fast most modern Markdown implementations are. Using an unmaintaned 5 year old perl script for benchmarking is ignorant.

</comment><comment><USER38> I remember when I moved off Wordpress to Jekyll, all my Wordpress stuff was in Textile using a Textile plugin. When I migrated off, I used a plugin that ran everything through the HTML rendered and exported it.

My new stuff is in Kramdown, and my old stuff is HTML with some Jekyll headers at the top. I did have to go back through some of it and clean it up too.

If I move to something new, I'd just render everything I have with Kramdown into HTML and move on to the next thing.

That's sadly how we have to deal with these engines.

</comment><comment><USER39> Seems similar to jade?

Edit expanding:

I liked jade but stopped using because of jsx, and disliking having to research solutions during unique use cases. I also have learned markdown fairly quickly, and like it more now, but I personally don't think it is as good of an experience reading plain text files as my own "plain text hacks" that I would use to format docs destined to be read in plain text. I feel that it clutters a nicely organized personal note, which adds to mental overhead.

</comment><comment><USER40> Does anyone think that there should be an alternative to markdown with a more natural syntax, that is text first? Something that is meant to be viewed in plain text most of the time, looks good in text, and easy to type in text, and has nice syntax highlighting.

Markdown isn't very aesthetically pleasing in plain text IMO. It's awkward to type, and it doesn't handle indenting the way people would normally create lists and indents in text. (## H2, ### H3 .. really??)

</comment><comment><USER41> I thought this is the famous Markdown parser by Vicent Martí. [1] It was the most popular Markdown implementation before CommonMark, and is used by GitHub (via RedCarpet, its Ruby binding) before they switch to CommonMark a while ago. The name choice is a bit unfortunate, but I guess it's too late to change it now.

[1]: [https://github.com/vmg/sundown](https://github.com/vmg/sundown)

</comment><comment><USER42> this defies what I want from a markup in plain text. It should look readable and formatted, even when not rendered.

Example: [https://raw.githubusercontent.com/gbraad/resume/master/resum...](https://raw.githubusercontent.com/gbraad/resume/master/resume.md) => [http://gbraad.nl/resume/resume.html](http://gbraad.nl/resume/resume.html)

I stopped reading after the first introduction of 'the syntax', as the intro either does not mention the why, or it totally misses the point.

</comment><comment> brian-armstrong A collection of Sundown documents could be called a Sundown Town [https://en.m.wikipedia.org/wiki/Sundown_town](https://en.m.wikipedia.org/wiki/Sundown_town)

</comment><comment><USER43> This is closer to an html templating language than anything else.

</comment><comment><USER44> I would rather consider it as an alternative to HTML.

</comment><comment><USER45> This has nothing to do with Markdown. You actually have to know html tags to use this.

</comment><comment><USER46> clicking thru on the signup link: [https://www.sunsed.com/sign- up](https://www.sunsed.com/sign-up)

Page not found. The requested URL /sign-up was not found on this server.

<comment><AUTHOR> Fixed it! Thank you for the heads up!

</comment></comment><comment><USER47> Someone needs to make a markdown-clone called "yadown?". Yet another down.

</comment><comment><USER48> How often will we reinvent LaTeX ?

</comment><comment><USER49> It feels like every other week there's a new alternative to Markdown..

Question is, do we need another alternative to the Markdown alternatives we already have?

<comment><USER50> It's a classic problem with markdown: it's a minimal markup language, which is ok for comments and super simple documents, but requires extensions for pretty much anything else.

This is why every week you see a new markdown implementation which is just ever-so-slightly different than the other, with every service and tool implementing it's own. I dread markdown, because I know there will be extensions. And the extension syntax doesn't look nice/simple. Syntactically, asciidoc or rst is much more thought-of than any markdown-like syntax I've seen.

This is also the main reason I really discourage markdown for anything _but_ a comment system. Just start with something more complete if you plan on documentation.

I was strongly against md on the documentation for julia for exactly these reasons. Not even one year later, md in julia is already contains extensions which are oh-just-very-similar-to-but-not-quite the others.

</comment></comment><comment><USER51> So you've created a website with your own name?

<comment><USER9> Would you please not be a jerk here? That's against the rules in general and particularly against the rules for Show HNs:

[https://news.ycombinator.com/showhn.html](https://news.ycombinator.com/showhn.html)

Edit: you've continued to post uncivil/unsubstantive comments for quite a while now. We ban accounts that do that. If you can't or won't fix this, we're going to ban you, so please fix this.

</comment></comment><comment><USER52> Goodspeed and may this ulterior specification become famous and used ; buuut i can't avoid thinking about a certain Xkcd strip [0]

[0]([https://imgs.xkcd.com/comics/standards.png](https://imgs.xkcd.com/comics/standards.png))

</comment><comment><USER53> There are many markdown implementations and standards. Madness! Lets create another one, but with a syntax that is worse somehow</comment>


Ask HN: Good Open Source Survey Software? - <AUTHOR>

I was wondering if anyone could recommend a good open source survey system, preferably in PHP.
I'd like one with a bit more intelligence than just counting up the answers, for example:
- For a question of "which of the following sites do you read (tick all that apply)", it would be nice to also be able to see statistics such as "x% read more than 3 of them" and "of people who read site A, x% also read site B"
- Tracking between answers: show results from question A made up only of people who also answered x in question D, etc.
Or would I be better with a paid hosted service?
Thanks <comment><USER2> <http://www.limesurvey.org/> should solve your problem, take a look , and I'll recommend you to host it on your own servers.</comment>


Startup = Idea - <AUTHOR> http://jmlite.tumblr.com/post/33443330774/startup-idea

<comment><USER2> A startup is whatever you think it is. There is no solid definition. I imagine most people will agree that a startup is always a newly formed business, which is probably geared to "grow" at a fast pace, and is probably not something traditional or brick and mortar.

The rest is just trying to figure out who you don't want calling themselves a startup.

(And by writing this I have joined the others in attempting to define this word.)

</comment><comment><USER3> If I'm starting a flower business, is that a startup? No. I'm just starting a business. Selling flowers is a proven business model.

The difference between starting a business or a startup is that a startup is trying to solve a unique problem. It often tries to do this through the use of technology.

If I think I might have found the cure for cancer which could lead to a billion dollar company but I need to do 5 years of research, that's a startup! No bank is going to give me a loan for this so I need investors who believe in my vision.

<comment><USER4> Uh, no. Legally, a startup is simply any new business, whatever goals the business is trying to accomplish. The startup period is the period during which the business is setting itself up prior to begin its normal intended business operations.

The term "startup" derives from the tendency of technology based companies to remain in this "starting up" period longer than companies in most other domains. It has somehow been romanticized as a term exclusively belonging to tech.

After all, if "startup" only meant a business trying to solve a unique problem, then Twitter, Facebook, and even Google never were startups, because communication, social networking, and search were not unique problems when those companies were formed. For that matter, AirBnb and Uber would not be startups, as FRBO's and charter car services are solved problems.

<comment><USER5> "Legally, a startup is simply..."

I normally dislike "citation needed" responses, but what law are you referencing, exactly?

<comment><USER4> For the tax code laws, see §195, 248, and 709. For the state and federal laws, I suggest using Lexis to look them up as i don't have the time to provide cites.

</comment></comment></comment></comment><comment><USER6> _> The next test: If a startup = growth then by definition a new business that is not a start up cannot = growth._

I think Paul Graham meant it to be a necessary condition to be a startup, but not an exclusive one. Pretending otherwise is just a camouflaged straw man.

</comment><comment><USER7> Start (New idea or approach) + Up (Scalable) = Startup (A scalable new idea or approach)

A new idea or approach that's not scalable would not be a startup (e.g. my uncle's underwater bar for scuba divers in Barbados). And a scalable idea or approach that's not new would not be a startup (e.g. another Coke/Pepsi clone).

</comment><comment><USER8> After reading that I still don't know the concrete definition of Startup. Is it an idea? I have many ideas that I have jotted down over the years. Are all those Startups? Or do I need to execute? The post suggests that if someone is building something that already exists then it is not a startup. Why? Is Hipmunk a startup? There already exists many travel search engines. You might say they have a unique spin on something that exists. Your hotel idea with a treadmill in every room had that as well.

<comment><USER4> There is a legal definition: a newly formed business that has not begun its intended normal business activities. Legally, and in the tax code, a "startup" has various advantages that established businesses do not (i.e., for example, in some states a startup does not pay income taxes for its first year).

If you want to add more to the definition...that is a philosophical battle that will never be resolved decisely for any specific extended definition.

</comment></comment><comment><USER9> Rant/shameless plug:

The author raises a few valid points -- which I would summarize as "any short definition of a complicated thing will have at least a few minor problems" -- but overall this feels pretty unconvincing to me.

For example, I see nothing wrong with seeing a new company aimed at creating a global chain of barber shops as a startup -- yet I see plenty wrong with describing his grow-vegetables-on-the-moon company as a startup (cool space technology + no obvious business model = scientific foundation perhaps, but not even a business, let along a startup).

Which makes it frustrating to see this on the front page, because I wrote my own response to PG's essay several weeks ago, [1] which I think was better thought out and written, and it got no love on HN (though it did on Twitter).

/rant /plug

[1] <https://news.ycombinator.com/item?id=4577525>

</comment><comment><USER10> I completely disagree with the theory "Startup = Idea".

My counter argument is: "Good Ideas Grow On Trees"

Working with a startup myself, I can clearly see why Paul Graham nailed it down to "growth" and not "idea".

The "idea" part of an startup is the easy one. I scientifically researched the topic 'innovation capability' and 'innovativeness' of organizations and the crucial part is not to come up with an idea. As said "good ideas grow on trees". The hard work is the execution of an startup idea to get "growth".

The main difference between startups is not their idea, but their growth. Many people can come up with a new startup idea in days, maybe even with a new idea within minutes. But it can take years to bring an idea to success.

Here are examples of good ideas that are growing on trees: [http://www.quora.com/What-is-the-best-way-for-a-first- time-e...](http://www.quora.com/What-is-the-best-way-for-a-first-time- entrepreneur-to-get-word-out-about-their-startup)

There are endless startup ideas. Even the idea newness, how radical new a startup idea is, is not the main aspect of an startup. Again the main difference between startups is if they can execute the idea and get growth. For example with our startup we were the first to hit a large, untapped or underserved market. But the lack of execution and growth was what killed it. There are always other people who sooner or later have a similar idea. The main difference again is which startup executes the idea the best, the fastest and gets the most growth.

<comment><USER7> Is Integrity Funding a startup? Founded in 2007, it has seen 12,443% growth over the past three years and is currently the 4th fastest growing company in the US.

Website: <http://www.integrityfunding.com/index.html> Reference: <http://www.inc.com/inc5000/profile/integrity-funding>

There are a lot of companies on that list of the 5,000 fastest growing companies in the US that I (and perhaps others) wouldn't consider "startups" at first glance.

Also, in your argument you reference your "startup" but say that the company failed because of "lack of execution and growth." But wouldn't, by your own definition, your company not have been a startup because, after all, it was just another idea that fell off the idea tree and failed to grow?

</comment></comment><comment><USER11> I do not think that this is correct. Ideas are a dime a dozen. I think that execution is the most important aspect of a startup. The idea might actually not matter all that much, indeed the underlying idea might change quite radically over the lifetime of the startup.

<comment><AUTHOR> The problem though is that (good) execution is also critical for all businesses. We're looking for a unique characteristic to identify a startup from "normal" businesses.

<comment><USER10> The unique characteristic is the growth.

Startups (successful startups) grow much faster than "normal" businesses.

Facebook, Groupon, Instagram, Pinterest, you name it.

</comment></comment></comment><comment><USER12> IMO a startup is a company trying to discover and prove a new business model or a radically altered existing business model.

</comment><comment><USER13> Startup = a business = an organization that swaps value for money.

<comment><USER14> like the Post Office?

</comment></comment><comment><USER15> The book "The Lean Startup" defines what a startup is. No need for further explanations. Go and read the book if you have not. Save yourself time and money with it.

<comment><USER16> PG is _at least_ as much of an authority on the topic of startups as Eric Ries so I think that's a bit simplistic.

I personally prefer the definition that Steve Blank offers: "a startup is an organization formed to search for a repeatable and scalable business model". This to me makes it clear where you should focus. Once you have discovered this model, your startup turns into a "regular" company and your priorities turn to operations and scaling.

<comment><USER17> Thanks for mentioning this. When I was reading this essay, my first thought was, "an essay that's seeking to define what a startup is that doesn't at least acknowledge Steve Blank's by-now-well-known definition just stumbled off the starting line."

<comment><AUTHOR> It was really focused in on PG's essays vs all essays written around defining a startup.

</comment></comment></comment></comment><comment><USER18> Startups as fast-growing companies take a very particular role in history, in economy and in a nation. This is a deep concept with a lot of ramifications and implications. Of course this is intrinsically linked to the concept of innovation and then new ideas but that sort of ideas and innovations which cause all the implication of fast-growing companies. This kind of companies are noteworthy in many scales since millennia.

For example, startups renew dying jobs, notably, at the nation scale.</comment>


RethinkDB: new documentation site, cookbook with examples, practical guides - <AUTHOR> http://rethinkdb.com/docs/

<comment><USER2> I've been playing around with Rethink for a couple of days now, and I just can't get over how nice the admin web UI is. Scaling up machines is quick and easy, and it's pretty simple to resolve master/secondary issues when one of the servers goes down (or it's shut off).

I'm seeing some performance issues, and I'm definitely going to spend some time messing with my test data and optimizing it for Rethink. Right now, a couple of basic counting queries take ~1 second when the MySQL equivalent takes milliseconds. The other issue I'm having is that I simply did an import of existing data from MySQL that contains dates, which are treated as strings, so I'll need to convert the dates into a native format (integer based timestamps) before I can get some more performance numbers.

<comment><AUTHOR> Keep us posted on the performance issues you experience-- the team hangs out on IRC (#rethinkdb on freenode) and are usually more than happy to help work through these problems. If you identify a specific query that isn't performant, please open an issue on Github as well ([http://rethinkdb.com/docs/comparison- tables/](http://rethinkdb.com/docs/comparison-tables/)).

Also, RethinkDB 1.8 (shipping next week) will include native date support. You can follow the progress and discussion of the planned interface here: [https://github.com/rethinkdb/rethinkdb/issues/977?source=cc](https://github.com/rethinkdb/rethinkdb/issues/977?source=cc)

<comment><USER2> Great. I've been meaning to jump on IRC with some questions, but haven't had the time to do so yet. I'll definitely be waiting for 1.8.

</comment></comment></comment><comment><USER3> Awesome to see. The docs have been pretty good even prior to this update, but it's nice to know they get some love in an open-source project.

(I'm preparing to launch a Go application w/ RethinkDB up back using the [community] Go driver: [https://github.com/christopherhesse/rethinkgo](https://github.com/christopherhesse/rethinkgo) — couldn't be an easier DB to use without an ORM)

<comment><USER4> I'm going to use same stack (and driver) for side-projects ;) really attractive DB and language.

</comment></comment><comment><USER5> Is RethinkDB now "production-ready"? I recall some people using it production, however I think the devs still don't call it production ready.

<comment><USER6> They track "production ready" with the following item in their tracker:

[https://github.com/rethinkdb/rethinkdb/issues/1174](https://github.com/rethinkdb/rethinkdb/issues/1174)

</comment></comment><comment><USER7> I've following this project for a few months now. Most of it really sounds to good to be true, so I waiting for someone to use it in a big application where reliability is important. Is there anyone using it in production for a big application? Would love to hear about it.

<comment><USER8> I guess it depends on what you mean by reliability - from the FAQ:

 _RethinkDB is not a good choice if you need full ACID support or strong schema enforcement — in this case you are better off using a relational database such as MySQL._

<comment><USER7> No.. I am not talking about full ACID support. Hardly any of the new object stores like MongoDB guarantee that in all conditions. I am talking about being generally reliable to store data and fetch data for a relatively important application on production.

</comment></comment></comment><comment><USER9> Since the RethinkDB guys are usually quick to answer here:

How is the JVM driver for RethinkDB going? Are there any ORMs that work well yet?

<comment><AUTHOR> There are a few community drivers worth noting:

\- kclay's Scala driver (rethink-scala): [https://github.com/kclay/rethink- scala](https://github.com/kclay/rethink-scala)

\- esycat's Scala driver (scala-rql): [https://github.com/esycat/scala- rql](https://github.com/esycat/scala-rql)

\- dkhenry's Java driver (rethinkjava): [https://github.com/dkhenry/rethinkjava](https://github.com/dkhenry/rethinkjava)

As far as ORMs go, there aren't any for Java that I'm aware of. ORMs for other languages can be found here: [http://rethinkdb.com/docs/frameworks-and- libraries/](http://rethinkdb.com/docs/frameworks-and-libraries/). There are several ORMs for JavaScript, Python, and Ruby so far.

We would love contributions for other languages. Shoot me an email at mike [at] rethinkdb.com if you're interested in building an ORM or community driver.

<comment><USER10> The Scala driver has some ORM features and I've been throwing around the idea of creating a java only version of the driver.. I guess you could use it now but would run into some of Scala odd method naming when trying to access from from java.

</comment></comment></comment><comment><USER11> Great work with the new docs! I think there's a missing comma before the secondary index object argument in the cookbook at "Efficiently retrieving multiple documents by secondary index":

      r.table("posts").getAll(1, 2, 3 {index: 'author_id'}).
        run(connection, function(err, result) {
            if (err) throw err;
            console.log(result);
      })

<comment><USER12> Thanks! I just pushed a fix for that.

</comment></comment><comment><USER13> Has anyone used RethinkDB with Heroku? I know there isn't an official add-on, but I was wondering if anyone had made it work. If so, I'd be interested to hear of anyone's experience, good or bad.

<comment><USER14> It looks like someone is working on building a service provider for RethinkDB: [https://www.rethinkdbcloud.com/](https://www.rethinkdbcloud.com/)

</comment></comment><comment><USER15> Awesome job, congrats!

Do you plan to implement automatic re-election in case of master failure?

<comment><USER12> It's on the roadmap -- See [https://github.com/rethinkdb/rethinkdb/issues/223](https://github.com/rethinkdb/rethinkdb/issues/223) to track progress.

There is no precise ETA for now though. Feel free to comment on the github issue if you need it. We build our roadmap on the feedback we get from our users : )

<comment><AUTHOR> I would also point out that it's included in the list of requirements before we declare RethinkDB production-ready (see [https://github.com/rethinkdb/rethinkdb/issues/1174](https://github.com/rethinkdb/rethinkdb/issues/1174)).

<comment><USER16> It's actually listed as a maybe there.

</comment></comment></comment></comment><comment><USER17> Quick word to say I really like the website.. congrats on the team behind it! : )

<comment><AUTHOR> Thanks! We'd love feedback from other people on usability, content, and design.</comment></comment>


Tim Sweeney claims that Microsoft will remove Win32, destroy Steam - <AUTHOR> http://arstechnica.com/information-technology/2016/07/now-tim-sweeney-thinks-that-microsoft-will-use-windows-10-to-break-steam/ <comment><USER2> The issue seems to me that app stores don't let other apps be app stores, maybe that should change.

I don't think Steam allow other game marketplaces to upload their software either, maybe that should change too.

</comment><comment><USER3> I think this is the first time I've seen "on the gripping hand" show up in an article. I wonder how many readers actually understand that?

</comment><comment><USER4> Microsoft ditching Win32 wouldn't just be sawing it's foot off it would be sawing it's head off.</comment>


No one ever got fired for using Hadoop on a cluster (2012) [pdf] - <AUTHOR> http://research.microsoft.com/pubs/163083/hotcbp12%20final.pdf <comment><USER2> More recent related project where a laptop outperforms clusters: [https://www.usenix.org/conference/hotos15/workshop- program/p...](https://www.usenix.org/conference/hotos15/workshop- program/presentation/mcsherry)

<comment><USER3> That's awesome.

On a meta-note, I felt like the abstract wasn't very clear and I had to skim the rest of the doc to figure out what was going on.

<comment><USER4> His blog posts are easier to follow and aren't forced to be in the academic format :)

[http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...](http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html)

[http://www.frankmcsherry.org/graph/scalability/cost/2015/02/...](http://www.frankmcsherry.org/graph/scalability/cost/2015/02/04/COST2.html)

<comment><USER3> Thanks!

No kidding. These arrangements are much more entertaining and informative.

</comment></comment></comment></comment><comment><USER5> Actually I know several projects bursted into flames because of Hadoop. There is a notion among some developers of Hadoop solving all of the problems and this is the only solution they consider (literary). Needless to say that real time systems with tight SLA cannot really use Hadoop, and if you are trying to sell it for this use case, you might get fired. :)

<comment><USER6> I concur and I have seen the same, especially for folks with complex projects, tight deadlines and budgets, and older distros of Hadoop that were not as stable and/or production ready.

In a rush to get to the next big thing, many have taken a lot of risk with Hadoop. Some with the resources to really support/make Hadoop their own have had a lot of success. But I think there are fewer who have success stories.

<comment><USER5> On the other hand, Hadoop is great for analytics. If it is down there is no customer facing outage. Sometimes the failover works and only few things lost, you are back in ~15 minutes if you want to check the fs image consistency. For internal teams this is acceptable SLA. If you don't care about consistency than you can probably do this faster.

I just don't see a customer facing SLA sensitive workload fitting this well. At least I would not sleep well being oncall... :)

<comment><USER6> I don't know that Hadoop is _always_ great for analytics in every case. Sometimes the overhead of spinning up mapreduces (for example) outweighs the cost of running analytics using other methods. Especially when you have demanding requirements for subsecond or milli-second response time to provide such analytics - I don't think Hadoop is there yet, although it's getting there.

Generally, I agree though, in an extremely demanding SLA environment... I probably wouldn't sleep well in that case either!

I think that was kind of the point of the paper as presented - Hadoop is seen as a panacea, when in reality there might be other, simpler approaches that work just as well or better. It really does depend on the use case, the volume/types of data, cost/requirements, etc.

For that matter, what the Hadoop ecosystem "is" (vs. just the Apache Hadoop project itself) means so many things now. HDFS (storage), YARN (distributed job/resource management), mapreduces, Hive, HBase, etc. vs. new engines, like Apache Spark, for example, which can run inside or outside of Hadoop. Adding to that the different distros and fragmenting Hadoop ecosystem, constantly changing versions, etc. - I don't know about you but it can be a nightmare even for analytics (in some cases).

When properly supported by knowledgeable staff with a deep grasp for what it can do as a platform, Hadoop can certainly be and do a lot of things for a lot of use cases.

<comment><USER5> I agree, I would not recommend Hadoop for a startup to install and maintain it. There are several datawarehouse as a service offerings out there. More recently there was a rise of good and cheap cloud based systems and Redshift got a lot better as well. Much easier to integrate, probably lover TCO too.

</comment></comment></comment></comment></comment><comment><USER7> This is just a bunch of FUD (fear uncertainly and doubt) that Microsoft spread back in 2012 when Hadoop didn't have Windows support. Get in our time machine and go to 2015, and Hadoop has Windows support. Suddenly Microsoft doesn't seem to mind it. Gee, I wonder why?

Hacker News is ridiculously susceptible to propaganda (MongoDB is WEBSCALE! JS framework of the month is the bee's knees!). Please think for yourself and don't read transparent propaganda that is years old. 3 years is an eternity in the big data world.

<comment><USER8> Seems like a valid point of view. Not sure why it got downvoted.

Though there are very big kinks to work out in getting Hadoop production ready.

So the thesis of the parent article is valid too and not just propaganda.

</comment></comment><comment><USER9> There are other systems which don't limit you to mad reduce patterns. For example HPCC.

<comment><USER10> Modern day Hadoop doesn't limit you either.

YARN containers are pretty generic and I have used them to run all sorts of things: Kafka, ElasticSearch etc.</comment></comment>


Ask HN: Is it OK or ethical to boycott movies, music if SOPA passes? - <AUTHOR>

I have been wondering about this question for quite sometime now. If SOPA or any of its cousins pass and Internet is, let's say, more than regulated; would it be ethical to boycott most of the entertainment that requires you to pay towards the Music and Movie industry establishments?
For: I am an Internet geek and I make my living coding. SOPA, PIPA etc. are a problem for me and my profession. Moreover, it is also about the most important source of communication and speech today. If the giants in the entertainment industry try to kill this medium, then it may be a good idea to boycott theirs.
Against: Artists, entertainers associated with those media giants who are backing SOPA etc. are genuine people with a job and are hard working. Like I won't like my job to be in jeopardy, I can't imagine them being in problem.
There could be more points but I hope I have the basic point across.
Thanks a lot. <comment><USER2> I think it's fair to do it, maybe then their own people will revolt against them. We could also try to get those people on our side by proving them laws like these won't really stop piracy, and they will only make them more dependent on the big labels, they will lose the opportunity to make some direction connections with their fans, and so on.

<comment><AUTHOR> Since, there are artists who have achieved heights began their careers on youtube, this can be pointed out to them.

There are others like Louis CK as well who may support us(?). Louis CK proved that the model he experimented with his last show worked. Didn't he?

</comment></comment><comment><USER3> You have no obligation to consumer anything.</comment>


Reasoning with Types in Rust - <AUTHOR> https://aaronweiss.us/posts/2018-02-26-reasoning-with-types-in-rust.html <comment><USER2> > A function of type fn<T>(T) -> T must:

> return its argument or

> panic or abort or

> never return

This confused me for a moment because I thought "why? it only says the return value must be the same type as the argument, not that it must have the same value", but of course since the type can be anything there is no operation that is guaranteed to work that you can perform on the argument value. You can't for example multiply by two because not all types will support multiplication by an integer.

<comment><USER3> Perhaps this is subverting the type-theoretic (in a CS sense) point being made, but in C++ you could quite reasonably write a function:

      template<typename T>
      T fn(T t){
          return t * 2;
      }

      fn(4.2); //fine
      fn("foo"); //fails

because the inability to multiply by 2 isn't a problem until an incompatible type parameter is instantiated. Is this not the case in rust?

<comment><USER4> That's because templates have no type-checking in C++.

Ideally, you wouldn't be able to do this with concepts, but I don't know whether that's true for the concepts that finally made it into the standard (i.e. I don't know if it would be an error to call a function or operator on an argument if that function isn't required in the concept specification.).

<comment><USER5> C++ templates have type checking after expansion, the type checking just isn't modular.

</comment></comment></comment></comment><comment><USER6> I loved this post. It's very readable and introduces some interesting concepts related to type safety and some potentially simple use case in the context of access to private data, guaranteed from a type pattern.

As was pointed out in the Reddit conversation about this [1], the parametricity (new word I learned while reading this last night), will be lost with specialization.

Personally I want specialization for all the joy it will bring, and I don't see this as a huge loss, but I wonder what other people's thoughts are on this?

[1] [https://www.reddit.com/r/rust/comments/80eis2/reasoning_with...](https://www.reddit.com/r/rust/comments/80eis2/reasoning_with_types_in_rust/?st=JE7ZUBNM&sh=5df7aace)

<comment> matt-noonan To me, parametric polymorphism is much nicer for reasoning about a program locally / modularly. It's kind of like reasoning about a function with linear control flow vs reasoning about a function with lots of nested if/elses. You can do the reasoning once and be done with it.

I tried to make a general point along those lines in this blog post, thought I think it deserves more extensive examples. [http://storm- country.com/blog/abstract-is-simple](http://storm-country.com/blog/abstract- is-simple)

</comment></comment><comment><USER7> The _Secret_ example is basically just encapsulation. It reminds me of the HTML type used in Go's standard library to avoid XSS vulnerabilities in HTML templates due to lack of escaping. [1] Any string that's not cast using HTML() is considered unsafe and will automatically be escaped.

In practice, this is used to make security audits easier. It's not that you can't defeat the system, but security review only needs to look at all the places a string is cast to or from the HTML type and verify that they're in fact safe. This is considerably easier than it would be with bare strings, though better encapsulation would help. (In Go, this could be done with a struct that has a private field, but that's not what they did for some reason.)

This is an old trick, also used by GWT [2] and other languages. Language support for zero-overhead, encapsulated wrapper types makes this particularly convenient; otherwise you have to decide if the performance cost is worth it for a widely-used type.

[1] [https://golang.org/pkg/html/template/#HTML](https://golang.org/pkg/html/template/#HTML) [2] [http://www.gwtproject.org/javadoc/latest/com/google/gwt/safe...](http://www.gwtproject.org/javadoc/latest/com/google/gwt/safehtml/shared/SafeHtml.html)

<comment><USER8> You can write parametrically polymorphic code in pretty much any language that has a concept of polymorphism [1]. It's not difficult to avoid doing the things that violate it. The problem arises in that if your language doesn't guarantee it, you don't know what third-party code will do. Go by no means guarantees parametric polymorphism, for instance, and technically a wizard armed with "reflect" can do anything they want, but in practice you can get a lot of mileage out of it even if it isn't guaranteed.

I write a lot of networking code and one of the patterns I use a _lot_ is to get a socket from something (connecting, accepting, whatever), do any socket- specific configuration I may need to do on the socket immediately, and then pass it to a function to do the real work that takes an io.Reader or io.Writer or whatever minimal interface it is I actually need. This has two major benefits: First, it makes it clear that the "real" code is only going to do certain things, and second, it makes the "real" code much easier to test because it's tedious and difficult to mock a real socket, but quite easy to pass something an io.Reader and ensure it does the right thing.

It's certainly not a new trick and it can be applied in a lot of existing languages, but as programming ideas go, it's still a relatively new concept to the programming gestalt community [2]. I expect to see this idea coming up more and more over the next 20 years; I wouldn't be surprised that on the cutting edge programming language side, it becomes as expected to have some form of enforced parametricity in your new shiny language as it is to have closures.

[1]: For the purposes of this discussion, dynamic languages have only one type, "Variant", and thus don't really qualify as polymorphic in this manner, despite their apparent extreme polymorphism.

[2]: Yes, it's a couple of decades old, but measured in person-years of understanding it is minuscule compared to the community as a whole.

<comment><USER7> I said it was basically the same thing, but a big difference is that we aren't using polymorphism, just encapsulation.

You don't need reflection to break parametricity. All you need is instanceof (in Java) or a type switch (in Go), and this is very common style since sum types don't exist and that's how we pattern-match. (Along with the visitor pattern.)

Parametricity seems to be when a language doesn't have these things; if you want to pattern-match on types, there needs to be a sum type or an operation that returns one.

It's not clear that programmers used to an object-oriented style will go along with the idea that downcasts are bad and should be forbidden. Is it worth removing a useful escape hatch (particularly in framework-heavy code) to gain an esoteric form of type-based reasoning?

But if you try Rust, you're fighting the borrow checker anyway, so perhaps this additional inflexibility isn't that noticable?

<comment><USER8> "You don't need reflection to break parametricity."

Correct; I cited reflect just because it lets you do _anything_ ; i.e., you can get a type you don't understand at all, and reflect can go groveling through the type to see if it can find an embedded type it does understand, or even directly muck about with unrelated struct values, or any other crazy thing. Type switching requires you to be able to name the types you're going to examine in advance; with reflect you can even do horrible things to types that didn't exist when you wrote the code.

But then, the answer is mostly not to do that. Go isn't academically pure, but neither is it full of people writing crazy code like that. And so, _technically_ , you can't use this reasoning in Go... but even so, enough of it is there that I get some mileage out of it. I find my expertise in Haskell transfers in Go in some bizarre and unexpected ways, because while Go doesn't have much, Haskell taught me a lot about exploiting what is there to the maximum I can. Even though by Haskell standards _haughty sniff_ Go is barely even a _programming language_ (well I never!).

</comment></comment></comment></comment><comment><USER9> Site isn't very usable on mobile.

<comment><USER10> Does your mobile browser support CSS Grid? If it does and there's still an issue, I'd love to fix it.

<comment><USER11> Grid doesn't have great support on mobile yet.

I'm sure you want to stick with it, and aren't really interested in something else, but Grid only started getting solid support across all the main mobile browsers in January. Lots of mobile devices are stuck on older versions, because of vendors not releasing updates.

<comment><USER10> It's been supported on iOS for just about a full year now according to caniuse. Unfortunately, they seem to lack meaningful historical info about Android, but assuming some kind of feature parity between Chrome versions on mobile and desktop, should've been about the same time.

I will definitely stick with grid, but it does suck that people have mobile devices that still haven't gotten these updates after a year. :\

</comment></comment></comment></comment><comment><USER12> Excellent fucking article.</comment>


"Wealth work" is one of America's fastest-growing industries - <AUTHOR> https://www.theatlantic.com/ideas/archive/2019/08/americas-hot-new-job-being-rich-persons-servant/595774/ <comment><USER2> New?

Dude, work has been by the rich, for the rich, for about as far back as I know of. Probably forever.

It's comparative advantage, right? You use Uber because you don't want to drive. You use Deliveroo because you don't want to cook. The supermarket workers earn 1/5th what you do. The restaurant staff earn 1/5th what you do.

And so on and so forth.

Anyone who makes the transition from working to middle class knows this intimately. Suddenly you're on the other side of interactions.

Is it problematic? Sure it's problematic. The entire economy is problematic - if everyone were upper middle class we'd melt the ice caps, faster than we do already (damn.).

<comment><USER3> > Dude, work has been by the rich, for the rich, for about as far back as I > know of. Probably forever.

And indeed the link is even more basic than that. Once you get past what you can spend to maintain a lifestyle, wealth is basically a measure of how much of societies resources you can call in, and to make more of it the better options are to organise people to do useful work.

It isn't a class thing that the rich organise the work. Organising people is approximately what makes a person rich.

If someone is organising large numbers of people and is poor ... that is more than a choice, they'd have to be working quite hard not to be bouyed into wealth. It is easy and natural to leverage followers into riches.

<comment><USER4> > Organising people is approximately what makes a person rich.

Which means, if we are to remove wealth inequality, we have to disincentivise everyone from organizing other people.

<comment><USER5> If an AI is doing it, maybe, but it also has to be unhinged from all of the people programming it or owning it. That was unthinkable until recently.

</comment></comment></comment></comment><comment><USER6> I think it's less of "wealth work" and really just "work" and it's good and necessary. Times are never constant, things change all the time. More traditional work (traditional in the context of the last 50-100 years) has been on the decline due to huge advancements in technology, globalisation and other reasons, so it's only normal that we see many jobs go, but equally and luckily also many new jobs come.

It's true that there is more "Uber for X" businesses out there today, but 30 years ago we also didn't have "Digital Agencies", "YouTubers", "Influencers", "SEO Experts", "Security Engineers", "Software Architects" and obviously "Uber Drivers". But today people have different jobs than 30 years ago and that is alright.

Personally I see this market grow even more in the next two decades. One day someone's job will be to stand in the queue for me to buy me tickets to Wimbledon tennis tournaments or a new iPhone during a launch event. Some might say this is degrading work, but others will love it. We might call it "wealth work" today with a bit of a negative meaning, but I bet a few hundred years ago people felt the same about someone who was cutting your nails, scrubbing your feet and giving you a head massage.

A great glimpse into the future of jobs is by looking at Japan. In Japan you can hire someone to "cuddle with you". It's not a sex worker, but as people feel more and more lonely, it's only natural to see a new market rise in this space - where there is a new need (and money) there is a new potential market.

<comment><USER2> > One day someone's job will be to stand in the queue for me to buy me tickets > to Wimbledon tennis tournaments or a new iPhone during a launch event.

This is always going to be degrading because it's literally an artificial meaningless thing. The operators could just do this for a fee, but they're forcing someone to burn their time instead.

<comment><USER7> >This is always going to be degrading because it's literally an artificial meaningless thing.

It's not really artificial, it's the product of having _first-come-first- served_ instead of _highest-bidder_ products.

>The operators could just do this for a fee, but they're forcing someone to burn their time instead.

Ignoring the choice of FCFS instead of highest bidder, this is just what work is. Someone is offering their time in exchange for monetary compensation.

<comment><USER2> It is precisely artificial.

Yes, it is work. Degrading work, because it's completely pointless.

I could pay someone to move the gravel on my drive from one side to the other over and over again for a year. Someone would probably accept that if I structured it as an 8-hour job with health insurance etc. It'd be the most mind-numbingly boring thing ever.

</comment></comment></comment></comment><comment><USER8> Alternative take: More people are able to start their own small businesses providing services because of increased total wealth. The wealth not only provides clients, but also provides increased access to capital to start said businesses.

<comment><USER9> That doesn't explain rising _demand_ for the services these businesses provide.

<comment><USER10> More affordable and convenient. A fitness trainer doesn't have to make their weeks worth of wages in a few hours if they have more clients in general. Also this kind of work no longer requires hiring an entire fronting agency, just a website ad and a cellphone.

</comment></comment></comment><comment><USER11> I think that "servant work" would be more apt in most cases. And it's sad, of course, that as a society, we don't have a better answer to this growing inequality.

</comment><comment><USER12> Conversely, "Not Wealth work" is one of America's slowest-growing industries.

I'm being very tongue-in-cheek, but as it turns out, the more you make, the more you spend. I view this as a great opportunity to supply a demand (or even better, _create_ a demand! $$$).

</comment><comment><USER13> and the reason they wealthy are flocking to the US? because they avoid having to pay for the social services programs to their employees which leads to huge profits. foreign workers comes to the US and because they know they can get all these social services once they've returned home, they can underbid their US counterpart. the expanding different between the haves and the have not is purely due to the world wealthy and foreign workers exploiting the lack of social services in the US.

</comment><comment><USER14> "wealth work" == servant

The only change is now the positions are less long term employer commitment and more gig work than ever.

</comment><comment><USER15> "Between 2010 and 2017, the number of manicurists and pedicurists doubled"

Lots of middle class women get manicures and pedicures. Besides, as hourly pay at the top rises, the rich are willing to pay more per hour for labor that saves them time. Economic growth is not a zero sum game.

<comment><USER16> Do you honestly believe the rich tip more if their wealth moves from 100 to 125 million?

<comment><USER17> I think you're forgetting that wealth trickles down. Extreme extra-strong emphasis on the "trickles".

<comment><USER16> I just don't know if that trickle rate increases as there's more to trickle. If you're already having all your desires fulfilled, why spend more?

<comment><USER18> Spending to satisfy your own desires directly benefits the buyer and the seller—that's the part people tend to focus on when discussing the "trickle down" effect—but what _doesn 't_ get spent on consumption ends up invested instead. That improves the productivity of labor, allows more goods to be produced at lower cost, and ultimately benefits everyone. Trade is good, but the greatest external benefit to society comes when one consumes less than one earns, thus leaving more for others. Even just stuffing the money in a mattress would benefit others by reducing the competition for goods and services; it's not as good as a profitable investment, but it's better than wasting resources on _mal_ investment.

<comment><USER16> Wouldn't that suggest that the interest rate should be something astronomical?

<comment><USER18> I'm not sure where you're getting a connection to high interest rates. I would expect just the opposite; if people have their desires satisfied and are looking to invest what's left over then their time preference will be lower (more willing to wait) and money should be easier to get (lower interest rates on loans).

<comment><USER16> I was thinking the real-world version of keeping it under the mattress would be in a savings account, which would be encouraged by high interest rates.

I'm confused as to why investing (I assume in the stock market) "improves the productivity of labor, allows more goods to be produced at lower cost, and ultimately benefits everyone" \- that money we invest in buying stocks, after the stock is initially bought from the business, is just shuffling chips around a table, it doesn't seem to be doing anything for anyone. Am I off base? How does a company see a direct benefit from already sold stocks?

<comment><USER18> Putting money in a savings account is still investing, which is why you're earning interest. What I meant by "stuffing the money in a mattress" was actually taking it out of circulation. I didn't meant to imply that this would be more profitable than investing, just that it would still be a net benefit to others since you're not using that money to bid up prices on things they might want to buy. Same amount produced plus less money in circulation equals lower average prices.

> How does a company see a direct benefit from already sold stocks?

Normally when you buy shares in the stock market, other than from an IPO or secondary public offering, the company doesn't directly benefit from that trade. The company received its benefit up front when it sold the shares. Of course no one would have _bought_ the shares if they didn't expect to be able to profit from them, either by selling to another investor or through dividends, so the fact that a market _will_ exist is critical to any company looking to raise capital by issuing shares to the public. A high stock price is also beneficial to any company looking to raise funds through a secondary public offering, and may translate into actual revenue for capital investment if the company holds some of its equity in reserve.

What I really had in mind, however, was not buying existing shares from other investors in the stock market but rather venture capital, IPOs, private equity, and the purchase of corporate bonds. In each of these cases the investment directly injects money into the company where it can be put to work enhancing productivity.</comment></comment></comment></comment></comment></comment></comment></comment></comment>


How Discord handles over a million requests per minute with Elixir's GenStage - <AUTHOR> https://discord.engineering/how-discord-handles-push-request-bursts-of-over-a-million-per-minute-with-elixirs-genstage-8f899f0221b4#.lb9vf1xt5 <comment><USER2> The most important part of this article is the concept of back pressure and being able to detect it. It's common in a ton of other engineering disciplines but especially important when designing fault tolerant or load balancing systems at scale.

Basically it is just some type of feedback so that you don't overload subsystems. One of the most common failure modes I see in load balanced systems is when one box goes down the others try to compensate for the additional load. But there is nothing that tells the system overall "hey there is less capacity now because we lost a box". So you overwhelm all the other boxes and then you get this crazy cascade of failures.

<comment><USER3> Yes. You need to adapt the capacity of the system to handle the full load with -N- boxes dead.

Corollary: If you have 2 boxes, each of them has to be able to handle all the traffic, so you can't save money by using smaller boxes :D

Corollary #2: If you have 2 datacenters, each of them has to be able to handle all the traffic, so you burn a lot of money :D

<comment><USER4> Instead of 2 boxes that can handle all your load, why not 4 that can each handle 1/3rd of the load?

<comment><USER3> Let's say that one box is down (don't care why, boxes die all the time).

You have 3 boxes, they're all running at 100%, or worse if the load is not perfectly balanced (it never is). You've got bad latency and you're up for cascading failures. Your system is in bad shape.

Now, supposing you have to deploy your application. Gotta take a server offline temporarily, the load will be 100% on the last 2 servers left... that can only take 66%. Booya! You're down :p

That's what happen when people are cheap on servers, in high performance systems.

<comment><USER5> Yes, but his point is still valid: You can to an extent reduce the overprovisioning you need by adding more, smaller units.

In your example, two boxes would have meant downtime no matter the capacity of each box too. All your example demonstrates is a reason why you might want to have capacity that allows taking more than one box offline....

Which could be handled with bigger servers, or more smaller servers.

<comment><USER3> Yes to all of this.

I write this to illustrate that you must have a significant batch of servers (about 5) in order to achieve costs savings (by using smaller servers) WHILE not affecting reliability.

Well, that is, if you care about rare cascading failures and performances issues.

</comment></comment></comment></comment></comment></comment><comment><USER6> Hate to be a party pooper, but I'd like to give people here a more generic mental tool to solve this problem.

Ignoring Elixir and Erlang - when you discover you have a backpressure problem, that is - any kind of throttling - connections or req/sec, you need to immediately tell yourself "I need a queue", and more importantly "I need a queue that has a prefetch capabilities". Don't try to build this. Use something that's already solid.

I've solved this problems 3 years ago, having 5M msg/minute pushed _reliably_ without loss of messages, and each of these messages were checked against a couple rules for assertion per user (to not bombard users with messages, when is the best time to push to a a user, etc.), so this adds complexity. Later approved messages were bundled into groups of a 1000, and passed on to GCM HTTP (today, Firebase/FCM).

I've used Java and Storm and RabbitMQ to build a scalable, dynamic, streaming cluster of workers.

You can also do this with Kafka but it'll be less transactional.

After tackling this problem a couple times, I'm completely convinced Discord's solution is suboptimal. Sorry guys, I love what you do, and this article is a good nudge for Elixir.

On the second time I've solved this, I've used XMPP. I knew there were risks, because essentially I'm moving from a stateless protocol to a stateful protocol. Eventually, it wasn't worth the effort and I kept using the old system.

<comment><USER7> I think you misunderstand the problem we are solving here. We are not trying to solve this because our system can't handle it. We are protecting it from when Firebase decides to slowdown in a way that causes data to backup and OOM the system. Since these are push notifications that have a time bound on usefulness we don't care about dumping to an external persisted queue like RabbitMQ or Kafka (we rather deliver newer notifications faster, than wait for the backed up buffer to flush). Firebase also only allows 1000 concurrent connections per senderId with 100 inflight pushes (that have not received an ack) which means that only 100,000 can be inflight. Ultimately if a remote service is providing backpressure because it is having a struggle no amount of auto scaling on your end is going to help you.

This service buffers potential pushes for all users being messages, that then watches the presence system to determine if they are on their desktop or mobile (this is millions of presence watchers and 10s of millions of buffered messages), and users are constantly clearing these buffers by reading on the clients and finally when a user is offline or goes offline we emit their pushes to them (which is what this article talks about). This service was evolved from our push system from the game we worked on and when it just did pushes only and no other logic it could push at 1m/sec in batches, but its responsibility has changed.

Context matters :)

<comment><USER8> Could you not reach pretty much the same result with a queue, though?

For example, workers could discard messages older than some threshold, quickly emptying the queue if there are expired messages. Clients might not even queue messages if the queue is currently too long, perhaps even providing a convenient signal for them to back off from their most chatty behaviour.

Some messages will not be delivered on time if there is significant backpressure. There is not much you can do about it, apart from avoiding choking yourself.

Perhaps the queue could work with a LIFO policy, to help at least some messages go through in time instead of having most messaged delayed near to the expiration threshold.

<comment><USER9> Erlang _is_ a series of queues.

<comment><USER6> You are hand-waving many properties of purpose-built queue systems.

<comment><USER9> Erlang _is_ a purpose built queue system.

</comment></comment></comment></comment></comment></comment><comment><USER10> Quick serious question: How does this company plan to make money? They're surely well funded[1], but what's their end game?

[1] "We've raised over $30,000,000 from top VCs in the valley like Greylock, Benchmark, and Tencent. In other words, we'll be around for a while."

<comment><USER11> They have an awful lot of information about video gamers in conversation history. They could mine that data for game companies and sell it as a way to help companies build better, more addictive and mechanically pleasing games.

<comment><USER12> We've adamantly stated many a time that we will never sell users data, or put ads in the app.

<comment><USER13> Isn't it a little irresponsible to make that claim while operating on VC funding with no monetization strategy?

I don't mean to imply that you yourself are being dishonest or that you will go back on your word at some point. I'm sure you have noble intentions and want the best for your users. But it might be beyond your control.

If the company gets sold or goes public, you could have drastic changes in management and their philosophy. You acknowledge this outright in your privacy page, under the "OUR DISCLOSURE OF YOUR INFORMATION" heading:

[https://discordapp.com/privacy](https://discordapp.com/privacy)

Discord has a policy in place that explicitly says they can do whatever they want with the information without limit, including selling it or transferring it without consent. This directly conflicts with employees' public claims about intentions.

The difference is that one of them is a legal acknowledgement end users must make before they use the software and the other is a 'feel good' thing to hear on someone's blog or forum post.

If they just wanted protection against an accidental slippage of data then that privacy page could be changed substantially. Instead, they pave the road for the explicit sale of data at a later time.

Or, if they wanted to leave the option open in the future they could say "This privacy policy is subject to change" and give users an opportunity to opt-out when it changes without historical data being subject to undisclosed future use. But this weakens their value in an eventual acquisition by someone who wants to monetize the data.

As it stands right now, in an eventual acquisition or even just some internal shifts of philosophy in the organization, all historical data is up for grabs for any potential use.

Discord really shouldn't have employees state that they will never sell users data when they explicitly allow and plan for that option. It may not be intentional dishonesty, but it comes close.

<comment><USER14> "The difference is that one of them is a legal acknowledgement end users must make before they use the software and the other is a 'feel good' thing to hear on someone's blog or forum post."

No, if it's from a known employee, it counts as a legal advertisement.

<comment><USER15> Only if that employee is a VP or above AIUI.

</comment></comment></comment></comment></comment></comment><comment><USER16> That's awesome and it just goes to show how simple something can be that would otherwise involve a certain degree of concurrent (and distributed) programming.

GenStage has a lot of uses at scale. Even more so is going to be GenStage Flow ([https://hexdocs.pm/gen_stage/Experimental.Flow.html](https://hexdocs.pm/gen_stage/Experimental.Flow.html)). It will be a game changer for a lot of developers.

</comment><comment><USER17> "Obviously a few notifications were dropped. If a few notifications weren't dropped, the system may never have recovered, or the Push Collector might have fallen over."

How many is a few? It looks like the buffer reaches about 50k, does a few mean literally in the single digits or 100s?

<comment><AUTHOR> Good question. We don't have metrics on the exact number dropped. We're using an earlier version of GenStage that doesn't give any information about dropped events. Once we upgrade we'll have a better idea.

<comment><USER18> There's another important question: How will the clients deal with the fact that they did not get a notification delivered? Will that mean they probably never receive a chat message? That could in some cases be catastrophic for the user. Or would it only mean that they may not get something instantly, which would not be too bad if the client would also poll the server or also try to catch up on notifications on reconnects.

<comment><USER19> When the push notifications hit FCM, Firebase do not guarantee delivery of those messages to clients (usually iOS or android devices). There are quite a few reasons that FCM/APNS might fail to deliver a message, so applications almost never have functionality depend on them.

As you say, you might not get the notification pushed to the device, but you should still see the message if you open the messaging app as normal.

<comment><USER20> This is indeed the case. Our real time system is outside of firebase and APNS and it handles the actual real time updates of chat state once the app is launched. We also have a delivery system that accounts for network cuts/switches and the like.

</comment></comment></comment></comment></comment><comment><USER21> "requests per minute" is such a useless unit of measurement. Please always quote request rates per second (i.e. Hz).

Makes me think of the Abraham Simpson quote: "My car gets 40 rods to the hogshead and that's the way I likes it!"

<comment><USER22> Here's a cool trick I figured out. If you have something measured in units per minute, you can divide it by 60 to get units per second. I won't even charge you to use the method even though I'm in the process of patenting it.

<comment><USER3> Actually. The conversion doesn't work.

The requests per minute number is an average.

The requests per second number should be given for peak load. That is a very important metric, a system has to be scaled to sustain the peaks, not the average.

We'd need to know the traffic pattern to know the multiplier, that is certainly not 60 :p

<comment><USER22> Units per minute gives you a lower bound on units per second. You can't reach an average of X/min without achieving at least (X/60)/sec.

</comment></comment></comment></comment><comment><USER23> 50k seems like a low bar to start losing messages at. If this was done with Celery and a decently sized RabbitMQ box, I would expect it to get into the millions before problems started happening.

<comment><USER7> These machines do more than just push. They also buffer messages for each individual user to "potentially" push if they don't read them on the desktop client. This happens before the flow this article talks about.

We currently have 3 machines doing this for millions of concurrent users. At the writing of this article it was 2 machines.

<comment><USER24> What size machines are these? I'm shocked that this volume is your max handling with Erlang unless your using a smaller T series AWS instance for this.

<comment><USER7> These are n1-standard8 on GCE.

These are getting easily over 30,000 requests a second each about updating queues for new messages. And also are subscribed to presence events from our presence system to millions of people. It is a very busy service ensuring we only deliver messages to people not at their computer.

<comment><USER24> So if they are delivering 30k a second per box and the max "backlog" you allow to build is ~50k, then you cap your backlog at under 2sec worth of delivery? Or am I missing something?

</comment></comment></comment></comment></comment><comment><USER25> I love Discord, and love Elixir too, so this is a pretty great post.

Unfortunate that the final bottleneck was an upstream provider, though it's good that they documented rate limits. I feel like my last attempt to find documented rate limits for GCM/APNS was fruitless, perhaps Firebase messaging has improved that?

<comment><USER26> It's not the final bottleneck, it's the first constraint. ;)

<comment><USER25> Hah, fair. It's always unfortunate when it's hard to address the real limitation though :)

</comment></comment></comment><comment><USER27> What is up with Discord? I feel like it's quietly (maybe not so quietly) one of the bigger startups to come out in the last two years.

It seems to have totally taken over a space that wasn't even clearly defined before they got there.

<comment><USER28> It does, doesn't it? I used to use Ventrillo, but then they screwed our small group out of our server connection. And we happily used Dolby Axon for a while. We tried Google Hangouts... for a while; until it just really didn't work well (it just disconnected and crapped out a lot). We tried using the Steam client's chat, but while ok for screensharing, it wasn't so great for chat.

But at some point we heard of Discord, which posed itself as a chat/vent replacement, started using it, and it _just works_. Which is huge, since the other stuff generally didn't (Axon was actually good).

<comment><USER29> Ventrilo is laggy compared to TS and Mumble. It won't show the lag as ms, but it is there, and it is real. Its due to the way the protocol works, or it is the server. It is no longer in development, and you can't even run your own server on your own hardware. The interface is from the 90s. You don't want to use Ventrilo for gaming in 2016.

TS supports plugins. No lag issues, can run on your own server. Closed source.

Mumble is open source, no lag issues. Interface is slightly less good than TS. Supports SSL.

Discord, like you say, Just Works (tm). It is very easy to use, the interface is amazing, its in active development, and setting up a server is free. It also works in the web browser.

If you're into Blizzard games, Battle.net recently added native VoIP in their client. The advantage that has as Blizzard gamer is you don't have to install any 3rd party software.

</comment></comment></comment><comment><USER3> I'd like to say that the official performance unit is the "request per second". And its cousin, the requests per second in peak.

The average per minute only gets to be used because many systems have so little load that the number per second is negligible.

</comment><comment><USER30> Anyone know of a equivalent libraries like GenStage for other languages? (Java, NodeJS, etc)

I'd definitely be able to put to use things like flow limiters and queuing and such, but none of my company's projects use Elixir :(

<comment><USER25> ReactiveX seems to have documented notions for it: [https://github.com/ReactiveX/RxJava/wiki/Backpressure](https://github.com/ReactiveX/RxJava/wiki/Backpressure)

Highly recommend the Reactive series of libs. They're typically very well done.

The guy below is right that Akka is perfectly suited.

</comment></comment><comment><USER31> I spend a lot of time in the PCMR Discord, which is pretty lively. The technology seems to be solid, while the UI has issues (notifications from half a day ago are really hard to find for example on mobile devices). Otherwise I'm on Discord every day and love using the service. I miss some slack features, but the VOIP is very good.

<comment><USER12> What features in particular? The most common one we hear is search, which is actually implemented and undergoing internal testing before a public preview soon.

<comment><USER31> It's just what I mentioned. I'll get a notification, and I just can't find where I was notified from. Like on Android, if I click on the notification I would expect it to take me to where the conversation happened where I was notified. It would take a really long time of scrolling to try and find the notification given the volume of discussion that happens. Can I just like click on something to see all my notifications from android, click on them and go to the conversation?

</comment></comment></comment><comment><USER32> million requests per minute, is this a big deal?

<comment><USER3> 16k per second. 83k per second during peak (assuming 80/20 default traffic rule).

\- 100 /s = typical limit of a standard web application (python/ruby), per core

\- 1.000 /s = typical limit of an application running on a full system

\- 10.000 /s = typical limit for fast systems (load balancers, DB, haproxy, redis, tomcat...).

\- Over 10.000/s You gotta scale horizontally because a single box [shouldn't] can't take it.

The difficulty depends on the architecture and what the application has to do (dunno, didn't go through the article). You make something that can scale by just adding more boxes, then it's trivial, just add more boxes. Well, it's gonna costs money and that's about it.

So no. Not a big deal at all... if you've done that before and you've got the experience :D

<comment><USER33> While 1k/sec seems to be an average throughput for most web apps due to all the logic, 10k/sec is nowhere near the limit for fast systems, many can do well into 6 figures per second with some now doing millions/sec.

<comment><USER3> Right. 10k is not a hard limit. It's the standard I expect, for real world applications, on classic server hardware, with limited tweaking.

The 6 figures benchmarks that send/receive 1 byte data with all unsafe flags enabled are not representative of real usage.

<comment><USER33> Your description for fast systems refers to high-performance software like load balancers and databases. In this case, 10k is nowhere near the limit on modern machines, they all do 6 figures per second.

<comment><USER3> With good hardware and good tuning, possibly. Again I'm not saying it's a hard limit, I'm saying it's a reasonable expectation for real world production usage ;)

At 100k requests/s on a load balancer. The HTTP headers alone (450 bytes) with zero content are more than 1300 MBps of traffic. And well, the CPU will bottleneck long before that amount of requests to parse.

</comment></comment></comment></comment></comment></comment><comment><USER33> Akka(.NET) or any actor system is a perfect fit for this and brings the same functionality to other languages and frameworks.

<comment><USER34> Not exactly. Without running on the BEAM you're left with cooperative scheduling (handing back control to the scheduler) of processes instead of pre-scheduling (the scheduler can stop you).

That makes it possible for one processor heavy operation to take over and slow down everything else. BEAM ensures that if you have millions of request coming through and suddenly 1000 4 day long operations kick off on the machine, that the millions of normal, smaller operations continue responding and performing as expected.

Fairly critical for the stability of real time systems.

The other piece here is that these processes are cheaper on the BEAM than any other platform in terms of RAM cost.

.5Kb / process on the Erlang VM. A goroutine in Golang is the next closest at 2kb.

The two combined are one of the big reasons why benchmarks don't tell the whole story with Erlang/Elixir. It's harder to measure consistency in the face of bad actors.

</comment></comment><comment><USER35> Is the number of Push Collectors to Pushers constant or can it vary based upon notification load?

<comment><USER20> It is constant - but iirc, it'd be trivial to make a dynamically scaling pool. At the end of the day, a pusher is just a TCP connection. Keeping a pool of fixed size and planning capacity around scaling horizontally is a perfectly acceptable approach - given you know the potential throughput for each pusher.

</comment></comment><comment><USER36> just wondering, what is the difference if I use two kind of [producer, consumer] message queues (say rabbitmq) instead of this? Does genstage being a erlang system makes a difference?

<comment><USER37> RabbitMQ is written in erlang. So basically you use it natively instead of bringing and configurating a big dependency. It just come with your language for free without needing another process, etc etc.

</comment></comment><comment><USER38> how does one achieve this in Celery 4? I remember there was a celery "batch" contrib module that allowed this kind of a batching behavior. But i dont see that in 4

</comment><comment><USER39> Why not use Kafka for back pressure?

</comment><comment><USER40> > _" Firebase requires that each XMPP connection has no more than 100 pending > requests at a time. If you have 100 requests in flight, you must wait for > Firebase to acknowledge a request before sending another."_

So... get 100 firebase accounts and blast them in parallel.</comment>