How Discord Scaled Elixir to 5M Concurrent Users - <AUTHOR> https://blog.discordapp.com/scaling-elixir-f9b8e1e7c29b <comment><USER2> This writeup make me even more convinced of Elixir becoming one of the large players when it comes to hugely scaling applications.

If there is one thing I truly love about Elixir, it is the easiness of getting started, while standing on the shoulders of a giant that is the Erlang VM. You can start by building a simple, not very demanding application with it, yet once you hit a large scale, there is plenty of battle-proven tools to save you massive headaches and costly rewrites.

Still, I feel, that using Elixir is, today, still a large bet. You need to convince your colleagues as much as your bosses / customers to take the risk. But you can rest assured it will not fail you as you need to push it to the next level.

Nothing comes for free, and at the right scale, even the Erlang VM is not a silver bullet and will require your engineering team to invest their talent, time and effort to fine tune it. Yet, once you dig deep enough into it, you'll find plenty of ways to solve your problem at a lower cost as compared to other solutions.

I see a bright future for Elixir, and a breath of fresh air for Erlang. It's such a great time to be alive!

<comment><USER3> I read it a little differently. The whole article is about how Erlang/Elixir fails at its core reason for existence (fast message passing between distributed processes) and all the complicated work-arounds they had to implement to avoid actually using this core feature of Erlang.

<comment><USER4> People tend to forget that scalability is not a binary property. You always scale up to some users, up to some architecture, up to some amount of nodes. There is no system that will scale to infinity without requiring developer intervention once business needs and application patterns start to settle in.

Distributed Erlang/Elixir _has known limitations_. For example, the network is fully meshed, which gives you about 60 to 200 nodes in a cluster. Or don't send large data over distribution, as that delays the other messages, etc. Some of those are easily solvable. For example, you can rely on your orchestration tools to break your clusters in groups. Or you can setup an out of band tcp/udp socket for large data. Others may be more complex.

The important question, however, is how far you can go without having to tweak, and, once you reach those roadblocks, how well you can address them. In many platforms, writing a distributed system is a no-no or, at best, they require you to assemble and tweak from day one. In this case, the ability to start with Erlang/Elixir and tweak as you grow _is a feature_.

And if you never run into those roadblocks, then you can happily continue running on the default stack. Just look at the many companies using Phoenix PubSub and Phoenix Presence, both distributed, without having to worry about fine-tuning the distribution.

<comment><USER5> What difference does it make? Erlang/OTP distribution doesn't have pluggable architecture. Sooner or later you will reach a point that you have to modify it. Then you are diverging from the original branch which makes it even more difficult to maintain it. You have to merge your additions into every release (minor or major) and test it thoroughly.

A better architecture for a distributed system has a strong composability property. It should be possible to modify every possible aspect of it on a running cluster without introducing downtime.

Write your own standard well documented distribution layer and become independent of underlying technologies.

<comment><USER4> What do you mean by architecture here? If you mean the roles different nodes take and their topology, I actually doubt you can be decoupled from your architecture in a distributed system because they directly affect your design and capabilities.

You can move away from fully meshed for topologies but how does this choice affect node ups and node downs? Rebalancing can affect how you store data in the cluster.

How many connections should you have between nodes? A single connection makes the ordering guarantee straight-forward. Multiple connections is more performance but requires care if you need ordering and is more efficiently done along side your application.

And what about process placement? On which side of CAP do you want your registries to sit?

If you and your team is capable of "writing your own standard well documented distributed layer" upfront, then you are in a better position than most to take those decisions. But writing a distributed system is hard, so I will gladly start with a well-designed system, especially at the beginning of the project, when it may be unclear which patterns I will need as my application and business grow.

And most times, it will be good enough.

As far as OTP goes, you can plug your own discovery/topology mechanism as well as your own module for handling the connection between nodes. But, as mentioned in my previous reply, some of those issues may be better solved on the side, e.g. a different tcp/udp connection for data transmitting.

<comment><USER5> I did also mention that for fast time to market it is indeed a good idea to use already available tools. By the way writing a distribution layer is not that difficult. Many companies/startups with scalable back-ends that are not using Erlang/OTP have already done it. The point is scalable software requires knowledge and Erlang/OTP is not going to magically solve it. But it seems many fans are trying to promote it such that it is a magic tool that is going to make a shit software become a hyper scalable one. Just look at the comments below that how people have gone crazy.

<comment><USER6> Erlang (and by extension Elixir) definitely provides a set of tools which are good for building highly concurrent distributed systems. And your systems are likely to have few errors as well as being resilient, if you know what you are doing.

But indeed, you need to know your tools like every other part of computer science. Storing 9 billion elements in an array, doing linear search and then complaining your linear search is too slow and needs to run faster will be disastrous to your architecture. Likewise, assuming communication is free is equally disastrous.

The problem with building a distribution layer in another language is the effort involved. Most of the companies which pull that off and get stability in addition are usually large, multinational, and has ample amounts of engineering resources to throw at the effort required.

<comment><USER5> Consider for some reason (either technical or political), this company decides to migrate their web socket servers to Akka/Rust/Go/NodeJS. Integrating these new servers into their core cluster is going to be deadly painful. They rely heavily on Erlang/OTP internal clustering. This is not even considered as a challenge in distributed systems but still really painful to implement because of their design decision. This is what they did wrong. Clustering and message routing part of the application should be technology agnostic.

<comment><USER7> That depends what you're going for. Moving web sockets elsewhere are pretty simple regardless of whether you're using Erlang/OTP or not because it's a frontal layer. If you do that all you are doing is creating another layer to hold connections that needs to relay to the rest of the application. The application behind it still has the responsibility to receive and relay everything from those web sockets, determine what message goes to who, etc.

You can move it out just the same as you can elsewhere, you just add an unnecessary layer of complexity and lose a lot of the capabilities already built into the language. It boils down to a simple question of, what would be the purpose behind that decision? What improvement would you get from deciding to move any of that elsewhere? Do you gain speed at the cost of complexity?

Clustering is a non-trivial problem. It can't happen naturally in any language that includes mutable data without explicit oversight and thus most of the needed tooling can't be built into the system and guaranteed to work everywhere. As soon as some part of the system goes from "send message, get response" to "send message, change variable referenced in memory on this machine, get response" you break your ability to naturally cluster.

That leads to a dependency in central relay points for things like setting up web sockets and then having code behind them communicate. The code behind them in whatever language is usually going to be talking to other central relay points like load balancers or pub sub databases rather than directly to other specific servers.

As soon as you have central relay points, they become something else that you have to monitor and scale even if they are very high volume.

There's nothing wrong with this approach and it will certainly work but if you want to avoid that and build a distributed system in another language, you've got to invest a lot in other areas.

<comment><USER5> > Moving web sockets elsewhere are pretty simple regardless of whether you're > using Erlang/OTP or not because it's a frontal layer.

It is supposed to be easy. But they are not using language agnostic messaging. Instead they rely on Erlang distribution protocol. This is what makes it difficult to integrate.

> what would be the purpose behind that decision?

1\. To achieve composability. I can replace one web socket server with a C++ implementation to see if I can handle 10 million connections per server. If I fucked it up return back to Erlang implementation otherwise fuck BEAM. I'm moving to native code. The beauty of it is that I can do it one step at a time. Even I can implement web socket layer in multiple languages and experiment all possible options in production and no one even notices it.

2\. I don't have to pay for 20 Erlang guys who are mostly senior devs. Get 5 of them to write the core and the rest of the team NodeJS guys.

<comment><USER8> You're acting like the distribution and serialization formats of Erlang are not open or documented.

If you're going for your C++ layer, try [https://github.com/saleyn/eixx](https://github.com/saleyn/eixx) for example.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> I'm continually impressed with Discord and their technical blogs contribute to my respect for them. I use it in both my personal life (I run a small server for online friends, plus large game centric servers) and my professional life (instead of Slack). It's a delight to use, the voice chat is extremely high quality, text chat is fast and searchable, and notifications actually work. Discord has become the de facto place for many gaming communities to organize which is a big deal considering how discriminating and exacting PC gamers can be.

My only concern is their long term viability and I don't just mean money wise. I'm concerned they'll have to sacrifice the user experience to either achieve sustainability or consent to a buyout by a larger company that only wants the users and brand. I hope I'm wrong, and I bought a year of Nitro to do my part.

<comment><USER10> A closed source walled garden chat service that survives purely on the free flow of VC capital 100% has no long term viability. No federation means as soon as the "next thing" shows up and wins way the VC dollars they'll disappear. It's so incredibly frusterating that so many companies and users make/support these closed environments even as we enter a new golden age of open sourced and federated technologies.

<comment><USER9> If there was an open source and federated equivalent to the features Discord provides I'd use it. There is no such product. Matrix is interesting, but the experience is no where near as polished as Discord and friends and that matters for mass adoption.

<comment><USER11> >If there was an open source and federated equivalent to the features Discord provides I'd use it.

But will you pay for Discord though? The features and quality Discord is able to provide are artificially propped up by VC funding. When it runs dry, we will be left with open source offerings, or the next product to take it's place and repeat the cycle.

<comment><USER12> I already pay $5/mo for Discord. Animated avatars and cross-server custom emoji were enough to entice me

</comment></comment></comment></comment></comment><comment><USER13> I know that the JVM is a modern marvel of software engineering, so I'm always surprised when my Erlang apps consume less than 10MB of RAM, start up nearly instantaneously, respond to HTTP requests in less than 10ms and run forever, while my Java apps take 2 minutes to start up, have several hundred millisecond HTTP response latency and horde memory. Granted, it's more an issue with Spring than with Java, and Parallel Universe's Quasar is basically OTP for Java, so I know logically that Java is basically a superset of Erlang at this point, but perhaps there's an element of "less is more" going on here.

Also, we're looking for Erlang folks with payments experience.

cGF0cmljaytobkBmaW5peHBheW1lbnRzLmNvbQ==

<comment><USER14> Off topic, but what's the story behind base-64 encoding your email? Is that just a spam prevention measure?

<comment><USER13> Yes

<comment><USER15> I actually really like this idea and am going to steal it. :)

</comment></comment></comment></comment><comment><USER16> Good stuff. Erlang VM FTW!

> mochiglobal, a module that exploits a feature of the VM: if Erlang sees a > function that always returns the same constant data, it puts that data into > a read-only shared heap that processes can access without copying the data

There is a nice new OTP 20.0 optimization - now the value doesn't get copied even on message sends on the local node.

Jesper L. Andersen (jlouis) talked about it in his blog: [https://medium.com/@jlouis666/an-erlang- otp-20-0-optimizatio...](https://medium.com/@jlouis666/an-erlang- otp-20-0-optimization-efde8b20cba7)

> After some research we stumbled upon :ets.update_counter/4

Might not help in this case but 20.0 adds select_replace so can do a full on CAS (compare and exchange) pattern [http://erlang.org/doc/man/ets.html#select_replace-2](http://erlang.org/doc/man/ets.html#select_replace-2) . So something like acquiring a lock would be much easier to do.

> We found that the wall clock time of a single send/2 call could range from > 30μs to 70us due to Erlang de-scheduling the calling process.

There are few tricks the VM uses there and it's pretty configurable.

For example sending to a process with a long message queue will add a bit of a backpressure to the sender and un-schedule them.

There are tons of configuration settings for the scheduler. There is to bind scheduler to physical cores to reduce the chance of scheduler threads jumping around between cores: [http://erlang.org/doc/man/erl.html#+sbt](http://erlang.org/doc/man/erl.html#+sbt) Sometimes it helps sometimes it doesn't.

Another general trick is to build the VM with the lcnt feature. This will add performance counters for locks / semaphores in the VM. So then can check for the hotspots and know where to optimize:

[http://erlang.org/doc/man/lcnt.html](http://erlang.org/doc/man/lcnt.html)

<comment><USER6> It isn't that likely the OTP20 optimization helps here. If the process never sends a message containing the literal value, then there is no benefit in the optimization. What `mochiglobal` and friends are good at is when you have a large set of data (A ring, say) which update rarely, so you can treat it as semi-static data in the system. But then you shouldn't really send that ring data around in the system too much, although it will now be free. [There is a nice subscription-based approach to updates which are now feasible in OTP20, but that is more for convenience]

if send/2 takes 30us to 70us, I'm guessing blocking as well, either on distributed communication or something else along those lines. For local message passes to take that long, my something-is-amiss-sixth-sense is tingling.

<comment><USER16> > It isn't that likely the OTP20 optimization helps here

Ah good point. I didn't look at the code much. I was thinking of cases of passing any of those literals in gen_server calls and such and just getting extra performance from upgrading to OTP20 as a side-effect.

</comment></comment></comment><comment><USER17> This is one of those few instances where getting the technology choice right actually has an impact on cost of operations, service reliability, and overall experience of a product. For like 80% of all the other cases, it doesn't matter what you use as long as your devs are comfortable with it.

<comment><USER18> Not sure why this comment saw a couple downvotes earlier. mbesto is correct: for most startups, most of the time, competitive advantage doesn't come from the underlying tech stack. To make a general statement, most things could be done similarly on any of several platforms. However, when product requirements match exceptionally well with a specialized technology, you can see things that would simply be infeasible or extremely tough using a different stack.

WhatsApp + Erlang was one of those cases (watch this talk and imagine trying to recreate that system with only a handful of server engineers using any other tech: [https://www.youtube.com/watch?v=c12cYAUTXXs](https://www.youtube.com/watch?v=c12cYAUTXXs)). Discord + Elixir appears to be another.

Curious if anyone has any examples that spring to mind from outside the highly concurrent messaging space.

</comment></comment><comment><USER6> A fun idea is to do away with the "guild" servers in the architecture and simply run message passes from the websocket process over the Manifold system. A little bit of ETS work should make this doable and now an eager sending process is paying for the work itself, slowing it down. This is _exactly_ the behavior you want. If you are bit more sinister you also format most of the message in the sending process and makes it into a binary. This ensures data is passed by reference and not copied in the system. It ought to bring message sends down to about funcall overhead if done right.

It is probably not a solution for current Discord as they rely on linearizability, but I toyed with building an IRCd in Erlang years ago, and there we managed to avoid having a process per channel in the system via the above trick.

As for the "hoops you have to jump through", it is usually true in any language. When a system experiences pressure, how easy it is to deal with that pressure is usually what matters. Other languages are "phase shifts" and while certain things become simpler in that language, other things become much harder to pull off.

<comment><USER8> The true evil approach is to send the socket around, not the message, so that there is no copying required no matter what ;)

<comment><USER16> Wah. Easy there, Satan :-)

That is cool trick though. So it's basically sending the port itself around and changing its ownership, with something like port_connect(Port,NewOwner)?

And btw, thank you for writing [https://www.erlang-in- anger.com](https://www.erlang-in-anger.com) and [http://learnyousomeerlang.com](http://learnyousomeerlang.com) !

<comment><USER8> The trick is more commonly used when writing to sockets. A socket owner is required for reading, not for writing.

The trick then is that when you need to write lots of data to a socket to just send a copy of it to the writer so they can dump all their data for cheap, but without changing ownership (which is costly).

Also recently I've gotten [http://propertesting.com/](http://propertesting.com/) out, you might enjoy it :)

<comment><USER16> Thanks for explaining. I'll have to remember the socket trick.

> Also recently I've gotten > [http://propertesting.com/](http://propertesting.com/) out, you might enjoy > it :)

It might be just what I need to understand and start using property tests. I've tried twice and gave up.

Oh and recon! Thanks for that too. Use it almost every day.

</comment></comment></comment></comment></comment><comment><USER19> According to Wikipedia, Discord's initial release was March 2015. Elixir hit 1.0 in September 2014 [0]. That's impressively early for adoption of a language for prototyping and for production.

[0] [https://github.com/elixir- lang/elixir/releases/tag/v1.0.0](https://github.com/elixir- lang/elixir/releases/tag/v1.0.0)

<comment><USER20> If I remember right, they were using Erlang at the beginning, and moved slowly to Elixir as they got more comfortable, and the ecosystem built up around them.

<comment><USER21> We started with Elixir, some of our engineers have used Erlang prior though.

</comment></comment></comment><comment><USER22> So, at this point, every language was scaled to very high concurrent loads. What does that tell us? Sounds to me like languages don't matter for scale. In fact, that makes sense, scale is all about parallel processes, horizontally distributing work can be achieved in all language. Scale is not like perforance, where if you need it, you are restricted to a few languages only.

That's why I'd like to hear more about productivity and ease now. Is it faster and more fun to scale things in certain languages then others. Beam is modeled on actors, and offer no alternatives. Java offers all sorts of models, including actors, but if actors are the currently most fun and procudctive way to scale, that doesn't matter.

Anyways, learning how team scaled is interesting, but it's clear to me now languages aren't limiting factors to scale.

<comment><USER23> > So, at this point, every language was scaled to very high concurrent loads. > What does that tell us?

Just like any language vs language debate each one has benefits for various particular use-cases. Any meaningful comparison of languages must be prefaced with the use-case scenario.

One of the strongest use-cases of Erlang/Elixir has always been building large distributed apps that need to scale (async web apps, telecom, chat servers, messaging mobile apps, etc). The ability to build these large distributed systems are baked into the very primitive parts of the language and standard library - to a degree that few other languages can compare to it, if any.

With Erlang/Elixir you design ALL applications in a way where scaling is rarely an after thought but rather a natural extension of the program.

> Beam is modeled on actors, and offer no alternatives. Java offers all sorts > of models, including actors, but if actors are the currently most fun and > productive way to scale, that doesn't matter.

People often make the mistake of trivializing Erlang/Elixirs as merely programming with actors. It's development not only predated the actor model but it also goes well beyond that to being the standard programming style you use when developing any program when using the language - the same way Rails embraces MVP. When this is fundamental part of every Erlang application then the means of scaling to a large distributed system are also a fundamental part of each program.

This built-in scaling is gained without any significant costs in terms of development time but also provides many benefits beyond scaling, such as highly modular and extensible code. There are real benefits even if you don't plan to scale to a large distributed system. similar to Rails it creates a predictable program design which makes joining new projects easier and deters NIH syndrome that is far too common in Java/C++/etc. And ultimately, regardless of what you are building, it provides very high performance by default for the type of async style applications that are popular on the web today.

So the key point here is not that the end goal was achieved (that you _can_ scale) but how you get there.

</comment></comment><comment><USER24> Great to see more posts like this promoting Elixir. I've been really enjoying the language and how much power it gets from BEAM.

Hopefully more companies see success stories like this and take the plunge - I'm working on an Elixir project right now at my startup and am loving it.

</comment><comment><USER25> Thanks for putting this writeup together! I use Elixir and Erlang every day at work, and the Discord blog has been incredibly useful in terms of pointing me towards the right tooling when I run into a weird performance bottleneck.

FastGlobal in particular looks like it nicely solves a problem I've manually had to work around in the past. I'll probably be pulling that into our codebase soon.

<comment><USER26> Note that Erlang 20 may have solved the problem that FastGlobal tries to fix (that of _not_ copying large amounts of data unnecessarily)

<comment><USER25> Erlang 20 fixes the case where you're copying a constant literal, but unfortunately won't help if you're sharing a dynamically generated, but infrequently modified, term; like Discord does in this post.

</comment></comment></comment><comment><USER27> Elixir was one of the reasons I started _using_ Discord in the first place. I figured if they were smart enough to use Elixir for a program like this then they would probably have a bright future ahead of them.

In practice, Discord hasn't been completely reliable for my group. Lately messages have been dropping out or being sent multiple times. Voice gets messed up (robot voice) at least a couple times per week and we have to switch servers to make it work again. A few times a person's voice connection has stopped working completely for several minutes and there's nothing we can do about it.

I don't know if these problems have anything to do with the Elixir backend or the server.

EDIT: Grammar

<comment><USER21> The messages struggles have been sadly due to issues with Cassandra and GC pauses caused by bugs within it. We have been trying to work with the Cassandra developers to resolve these.

Voice issues should not be happening. Please contact our support with more information and we will gladly investigate.

<comment><USER27> Thanks for the response, it's good to know what's causing the problems with messages and that it's being worked on. I'll try to contact support next time I have voice issues with my group.

<comment><USER21> :) np

We love Cassandra, and hate it at the same time.

Check out this nasty bug we got. [https://issues.apache.org/jira/browse/CASSANDRA-13004](https://issues.apache.org/jira/browse/CASSANDRA-13004)

<comment><USER28> Two things are true about Cassandra:

It is by far the best at doing what it does

It has plenty of room for improvement

<comment><USER21> You are 100% right that it is the best at what it does.

And thank you again with helping us with that bug :) we really appreciate it.

<comment><USER29> I'm currently far down the database rabbit-hole and have to ask: What's so great about Cassandra that you can't get with CouchDB or other AP (yeah, I know...) databases?

<comment><USER28> Solid ingestion story. Very very good write throughout. Linear scaling. Easy expansion / contraction. Complete flexibility in consistency vs availability tradeoff.

And most importantly:

It actually works at scale. Huge scale. Thousand node cluster and hundreds of thousands of instances scales.

Because a good chunk of the active maintainers actually run this shit in prod.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER30> I do not see there any Elixir specific, it is all basically Erlang/Erlang VM/OTP stuff. When you using Erlang, you think in terms of actors/processes and message passing, and this is (IMHO) a natural way of thinking about distributed systems. So this article is a perfect example how simple solutions can solve scalability issues if you're using right platform for that.

<comment><USER31> You're right. Elixir doesn't pretend to do anything except make using the wonderful erlang VM/OTP stuff easier.

The VM is an absolute marvel of engineering, and it's insane to me that it doesn't have more adoption yet in big tech companies.

My best theory is that engineers in top engineering companies are actually not the best engineers but simply career engineers that learn one skill (python/java/C++) and then explain to every employer that this technology is the best for the problem they have, over and over again.

<comment><USER32> Coordination problems get more difficult in larger teams/companies. Getting everyone to use a particular non-standard language is a coordination problem. Thus large companies are unlikely to experiment with languages.

It makes sense too - say it's 10x easier to write something in language X than Y. If there's only 10 other people that might interact with the thing / have to read the sources, that's a great tradeoff. If there's a thousand other people that might have to at some point understand how some part of your code works, suddenly all of them have to learn the new language X.

</comment></comment></comment><comment><USER5> It seems awkward to me. What if Erlang/OTP team can not guarantee message serialization compatibility across a major release? How you are going to upgrade a cluster one node at a time? What if you want to communicate with other platforms? How you are going to modify distribution protocol on a running cluster without downtime?

As soon as you introduce standard message format, then all nice features such as built-in distribution, automatic reconnect, ... are almost useless. You have to do all these manually. May be I'm missing something. Correct me if I'm wrong.

For a fast time to market it seems quite nice approach. But for a long running maintainable back-end it not enough.

<comment><USER33> The erlang team has been through 20 major releases, I think I've run all of them since r13 or r14? Being able to upgrade your distributed system is important and they care about making it possible. Generally you upgrade one node at a time, until they're all upgraded and only then can you use new language features. Sometimes you find a build you like and stick with it until something comes up that causes you to upgrade.

There are many ways to interoperate with other languages. Including libraries for other languages to claim to be erlang nodes (they will have to upgrade too when you want to upgrade to a newer version of erlang with distribution protocol changes, for example when maps we're introduced in r17).

You can easily do standard dist messaging within your erlang cluster and serialize to whatever makes sense at the boarder. You can serialize to erlang term format, if you like; it's well specified, but not terribly compact.

You're going to have the same questions with any other language too. Very few companies get to write clients, servers, and everything else in one language that never updates.

<comment><USER5> > You're going to have the same questions with any other language too. Very > few companies get to write clients, servers, and everything else in one > language that never updates.

Then I don't see much difference between Go, Java, ... vs Erlang except they are simpler to learn plus finding Java, NodeJS, ... devs is much easier. What was the point of using Erlang? It was supposed to solve a problem for us, but we end up of solving the problem ourselves.

What I'm really trying to say is: integrating a mainstream language like Go, Java, C++, ... with a messaging layer like ZeroMQ (or something else) and adding some reliability features is going to be easier than introducing a totally new language (with a totally different paradigm) into the stack.

<comment><USER33> > Then I don't see much difference between Go, Java, ... vs Erlang except they > are simpler to learn plus finding Java, NodeJS, ... devs is much easier.

For the first several years, my team of Erlang devs had zero Erlang experience prior to being hired; they were just smart and flexible developers. I was ahead of the curve because I sort of remembered seeing a Slashdot post about Ericson opening the language way back when.

A lot of people end up using RabbitMQ as their distributed message queue, which is built in Erlang. If you went with that instead of ZeroMQ; and then slowly added more things, there's a reasonable path to writing more Erlang.

I'm not sure if you can really bolt on messaging and reliability features and get the same results; just like bolting on security later, if it's something that you need, it works better if you have it from the beginning.

But certainly, if you're happy with your stack, don't change it.

</comment></comment></comment></comment><comment><USER34> Really liked the blog post. Elixir and the capabilities of the BEAM VM seems really awesome, but I can't really find an excuse to really use them in my day to day anywhere.

</comment><comment><USER35> Whatsapp's story is somewhat similar. Relevant read to this subject.

[http://www.erlang- factory.com/upload/presentations/558/efsf2...](http://www.erlang- factory.com/upload/presentations/558/efsf2012-whatsapp-scaling.pdf)

</comment><comment><USER36> I love discord's posts they are very informative and easy to read.

</comment><comment><USER37> 5 million concurrent users is great and all, but it would be nice if Discord could work out how to use WebSockets without duplicating sent messages.

This seems to happen a lot when you are switching between wireless networks (E.g. My home router has 2Ghz and 5Ghz wireless networks) or when you're on mobile (Seems to happen regularly, even if you're not moving around).

It's terribly annoying though and makes using the app via the mobile client to be very tedious.

<comment><USER38> Coming from IRC with netsplits and stuff, this seems like a first world problem to me hehe

<comment><USER37> You know there are discord bots for bridging between IRC and Discord?

Pretty funky :-)

</comment></comment></comment><comment><USER39> I really like elxir the language, but find myself strangely hamstrung by the _mix_ tool. There is only an introduction to the tool, but not a reference to all the bells and whistles of the tool. I'm not looking for extra bells and whistles, but simple stuff like pulling in a module from GitHub and incorporate it. Is there such documentation? How do you crack Mix?

<comment><USER40> The docs for Mix are decent. You can start here:

[https://hexdocs.pm/mix](https://hexdocs.pm/mix).

When you are trying to get help w/ a specific task, you check checkout the mix tasks docs:

[https://hexdocs.pm/mix/Mix.Tasks.Deps.Get.html#content](https://hexdocs.pm/mix/Mix.Tasks.Deps.Get.html#content)

And honestly, I often just look at the source code:

[https://github.com/elixir- lang/elixir/tree/master/lib/mix](https://github.com/elixir- lang/elixir/tree/master/lib/mix)

The Elixir Slack channel is pretty amazing, too:

[https://elixir-slackin.herokuapp.com/](https://elixir-slackin.herokuapp.com/)

</comment></comment><comment><USER41> It looks like they have built an interesting, robust and scalable system which is perfectly tailored to their needs.

If one didn't want to build all of that in house though, is there anything they've described here that an off the shelf system like [https://socketcluster.io](https://socketcluster.io) doesn't provide ?

<comment><USER42> Yes. Discord is served over HTTPS. :P (your link is broken; socketcluster.io doesn't serve over HTTPS)

But seriously, Discord actually benchmarked 5 million concurrent users, horizontally distributed, and having to ferry messages across the cluster, with specifically tailored fanout patterns (rather than just a global pub/sub. I.e., who a message goes to varies, rather than just "everyone").

Socketcluster.io only has benchmarks for a single machine, capped at 42k concurrent connections (though to be fair that was due to them running a single client, rather than a limitation of the server). They don't out of the box support horizontal scaling; you're required to spin up your own message queue solution for that.

So, basically, you're advocating a technology that solves -the simplest part of the problem-, and nothing else. Whereas Phoenix + Elixir, even without any of the custom tweaking Discord describes, solve that AND more of the actual problem Discord had. So...yes, and no. Yes, there is plenty here they've describe that is not available in socketcluster.io, but no, nothing they've done here is no generally solved by an off the shelf system, because they're -using- an off the shelf system, Elixir + Phoenix.

<comment><USER43> Hi, main author of SocketCluster here.

SC does support automatic horizontal scaling across any number of machines out of the box if you're running it on Kubernetes.

There's also a CLI tool to deploy it automatically to any Kubernetes cluster: [https://www.npmjs.com/package/baasil](https://www.npmjs.com/package/baasil)

See [https://github.com/SocketCluster/socketcluster/blob/master/s...](https://github.com/SocketCluster/socketcluster/blob/master/scc- guide.md#scc-guide)

<comment><USER42> I just meant you still need a third party MQ to be spun up (per docs here - [http://socketcluster.io/#!/docs/scaling- horizontally](http://socketcluster.io/#!/docs/scaling-horizontally)). Without that, there is no distribution happening.

From my understanding, you're basically saying "You can combine SocketCluster with the MQ of your choice (the installation and configuration of which is left as an exercise to the reader) and then between Docker, Kubernetes, and Baasil you can orchestrate and deploy it across a cluster". That sounds a bit more complex than just using SocketCluster, which is what the OP seemed to be indicating was all you needed, and is also including the DevOps story, which I don't think either he or I was intending to include.

I was not trying to indicate that SocketCluster can't be -used- to scale websockets horizontally, but that it's not just an off the shelf solution that would have solved Discord's problem either. It requires other parts, as both the docs and you mention.

I'll also reiterate from my post, SocketCluster has no benchmarks pertaining to what happens when you -do- scale horizontally (per docs here - [http://socketcluster.io/#!/performance](http://socketcluster.io/#!/performance) ). That lack alone would kill my interest in it (as would scc-state being a single instance, which would make fault tolerance a real concern to me, but it looks like you know that already). Is performing horizontal scalability tests on the roadmap?

<comment><USER43> If you use SCC, then you don't need a separate MQ - That is only if you want to do things yourself manually. I will update the docs to make that clearer.

It should only take a few minutes to deploy a cluster across hundreds of machines. The only limit is the maximum number of hosts that Kubernetes itself can handle (which is I think is over 1000 now)? SCC is self-sharding and runs and scales itself automatically with no downtime.

You can easily handle 5 million concurrent users with a small cluster. SC's problem isn't scalability, it's marketing.

<comment><USER42> That's perfectly fair; fix the marketing then. :P In evaluating a solution, the marketing is the -first thing- anyone looks at. And how it currently reads, "SocketCluster only provides you the interface for synchronizing instance channels - It doesn't care what technology/implementation you use behind the scenes to make this work" definitely reads as "You need a technology/implementation behind the scene" rather than "we provide you a default one, and you can swap it out".

For me to pick Socketcluster for a distributed solution (or more broadly, what I'd want for -any- technical solution) I'd want to know what else I need to pair it with (which the docs actually mislead me on), what else I can benefit from (which the docs don't tell me, but which does exist per your links), and what benefits I stand to get from using it (the docs tell me only marketing claims, but with no metrics, performance, data, etc, for what happens in a distributed context, well, I would avoid it).

Ideally, set up a clustered performance test, and then make as many of the artifacts (docker images, configs, readme, etc) available so others can conduct the same performance test themselves (as well as have a reference architecture for their own solution). Heck, if you're doing it in AWS, consider making the AMIs available along with whatever modifications need to happen. -That- would be very convincing for someone looking to adopt a solution in this space, if they could literally just spin up some EC2s and immediately start throwing load at a fully configured cluster.

Also, to make it clear, is this handling message passing between instances in the cluster?

<comment><USER43> Thanks for the advice.

Yes, it handles message passing between instances in the cluster. That means if you publish a message on a channel whilst connected to one host, the message will also reach subscribers to that channel which are on any other host in the cluster. It shards all channels across available brokers, when you scale up the number of brokers, it will automatically migrate the shards across available brokers with no downtime.

<comment><USER42> Okay, nice. Then yeah, given some performance benchmarks showing some numbers at increasing number of nodes, with messages being broadcast across 1-to-1 channel pairings (i.e., direct message), 1-to-100 (for groups), and 1-to-all, I think it could sell itself as a pretty compelling turnkey solution (barring the scc-state concern which you're aware of).

Possibly also consider some testing and documentation around geographic distribution; what happens if the nodes are located in different datacenters with non-trivial latency between them? Is that an issue? In the event of netsplits, does it split brain (probably not, given scc-state, but addressing that might cause it to)? That might be fine, it might not, depending on the use case, and just documenting what happens (by default, at least, if it's to be tunable) would be helpful as well.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER44> Reading posts like this about widely distributed applications always gets me interested in it as a career path. Currently I'm working as a front-end dev with moderate non-distributed back-end experience. How would someone in my situation, with no distributed back-end experience, break in to a position working on something like Discord?

</comment><comment><USER45> I think while this is great, it is good to remember that your current tech stack maybe just fine! after all, Discord start with mongodb[0].

[1]. [https://blog.discordapp.com/how-discord-stores-billions- of-m...](https://blog.discordapp.com/how-discord-stores-billions-of- messages-7fa6ec7ee4c7)

</comment><comment><USER46> Is there any update on BEAMJIT?

It was super promising 3 or so years ago. But I haven't seen an update.

Erlang is amazing in numerous ways but raw performance is not one of them. BEAMJIT is a project to address exactly that.

[https://www.sics.se/projects/beamjit](https://www.sics.se/projects/beamjit)

<comment><USER6> Still ongoing work. My personal bet is a bit more on modernizing HiPE however (by using the LLVM backend more).

<comment><USER46> Amy ETA on when we can start using beamjit?

<comment><USER6> Given that it has been postponed a couple of times, no. JITs are hard to pull off and it will probably have a period of worse stability as well before it matures. Another problem is getting a JIT to be faster than the interpreter. Erlang's BEAM is threaded code and also macro-instructions, so it almost looks like a JIT internally.

The big gains would be in inlining across module boundaries and type speculation. But I hold that if we could compile bundles of modules in HiPE, we would have the same gain for a fraction of the development and maintenance effort.

The biggest lure of native code generation would be that we could get rid of a lot of C code in the system as the native cogen would be able to rival the C code in speed. Many Erlang programs spend shockingly little time in the emulator loop, especially if they are communication heavy.

If you need speed today, don't underestimate a port-program. My test is that you can pipeline about a million requests back and forth to an OCaml program per second per core. So if your work is on the order of 1+ milliseconds, this is usually a feasible strategy. Espcially because OS isolation means you can handle exceptions in the OCaml program from the Erlang side by restarting the port.

<comment><USER47> Would you say the OCaml-port-as-an-optimization-strategy only makes sense if the program is compute bound, though?

The reason I ask is because we're running a Erlang+JInterface program and the performance advantage the JVM has over BEAM is less than I would have expected. Even batching requests up into big pieces, we still see it's about 30% slower than running the same stuff in Elixir, without so much copying. But the reason we're doing JVM stuff at all is so we can re-use a whole bunch of code we already had written, and I would have expected it would have been a marginal performance win as well, but it's not.

Perhaps we're doing it wrong, too.

</comment></comment></comment></comment></comment><comment><USER48> Very interesting article! One thing I'm curious about is how to ensure a given guild's process only runs on one node at a time, and the ring is consistent between nodes.

Do you use an external system like zookeeper? Or do you have very reliable networking and consider netsplits a tolerable risk?

<comment><USER49> We use etcd.

</comment></comment><comment><USER50> It's interesting how on StackOverflow Jobs Elixir knowledge is required more often than Erlang.

[http://www.reallyhyped.com/?keywords=erlang%2Celixir](http://www.reallyhyped.com/?keywords=erlang%2Celixir)

<comment><USER51> Despite its appeal to HN geeks I doubt if Elixir will ever achieve mainstream adoption. Searching Indeed.co.uk's API by title, there are only 5 Elixir jobs in London, compared with 445 Python and 171 Ruby. I also attended a Silicon Roundabout jobs fair recently and was disappointed to find Elixir wasn't even listed in the literature.

<comment><USER52> It's still early. I've bet on the wrong horse (stack) before, so take my comments with a grain of salt, but I know several companies that are currently adopting Elixir/Phoenix with the same level of excitement that I recall from the early days of Ruby/Rails. It may never be "Rails big", but there is definitely some momentum building.

<comment><USER53> When Rails arrived on the scene it was very different from everything else, but also there weren't 1000 new languages/ frameworks popping up all at once.

I'm not implying in anyway that Elixir is bad. I just think there are too many horses these days to know which to bet on. Elixir? Node? Go? Rust? Something else?

<comment> vertex-four Depends on what you're doing, really.

Rust is lovely for security-sensitive code - I'm writing a customer identity management system with Rust+Postgres+Redis.

Go is... acceptable... for "glue" code where PHP would've been used a decade ago and Perl before that - all the successes of it I've seen fall into that sort of pattern.

Elixir is great when you need to think about networked, stateful systems on the scale of a rack of machines - it provides many of the components to help you design systems at that scale.

So... as ever, they all do quite seriously different things. I don't think many people need to build the sorts of systems Elixir is good for - it'll always have its niche in large-scale communications systems, though. A good many webapps fall into the patterns that Go is good for - take user input, munge it, send it to some backend system. A fair bit of code that drives the foundations of what those webapps are built on will eventually be written in Rust.

<comment><USER53> I understand where they are supposed to fit, but the problem is there are so many existing tools to fill the same needs already.

As an example Erlang and Elixir both fit essentially the same bill so it seems to me what was already a small niche is just getting fractured.

<comment> vertex-four Elixir is just alternative (Ruby-ish) syntax for Erlang, as much as people have hyped it up - Erlang code can call Elixir code and vice versa with essentially no abstraction cost. In fact, the most popular web framework for Elixir is heavily built on Erlang code.

<comment><USER53> That was my point exactly. Before if you really needed the benefits of running on BEAM then you would have chosen Erlang. Now that subsection will fragment between Erlang and Elixir. Today Elixir developers still will lean heavily on the interaction of Erlang libraries until someone in Elixir-land needs something that isn't supported so then they will re-invent that particular wheel in Erlang and thus the cycle continues ever on.

Then as you stated yourself Go is being used in areas where PHP & Perl were used before (as were also Ruby, Python, and Lua). So now there is just one more fracture there. Of course this is especially interesting since Go was supposed to be a better C or C++ then that should have placed it pretty squarely in systems programming land, yet is has managed to gain more traction as a glue and web services languages, ergo it ends up competing in this space when it probably should have competed more with C/C++/Rust.

Rust; as a systems programming language still has to compete with C/C++/ObjC in this space and considering that "all the systems" already run on these languages that is one huge challenge to change the guard there. I'm not saying that Rust is bad, rather just that I think it's long term outlook may actually be rather bleaker than other languages due to these challenges. Quite frankly D had some great concepts and improvements as a systems language over C++, but it is still barely a blip on anyone's radar.

If an individual is just trying to learn and expand their horizons then any of these languages are great to pick up. If, on the other hand, they are banking their future career on one then none is a sure fire bet right now. One would honestly still have more luck with a tried and true like Java.

Personally I enjoy picking up new languages just to see things from a different perspective and continue learning. Heck I spend much of my free time coding in Crystal which hasn't even hit a 1.0 launch so that adds pretty much zero improvement to my career prospects. :)

<comment> vertex-four I wouldn't bank a career on a language anyway, so.

However, as Elixir is just another skin for Erlang, switching from Elixir to Erlang or LFE or whatever the next language for that runtime is will be simple - the issue is really BEAM and OTP, as few of the skills learned around them are cleanly transferrable to other runtimes and frameworks.

I suspect that Go will be around for a while by sheer force of inertia at this point - we're 5 years since 1.0 now and it still seems like there's new major projects being started in it every day.

Rust is... interesting, but I think it'll always have its niche. As much as people like to riff on the RIIR crowd, there's actually a somewhat decent ecosystem of reimplemented libraries and systems in Rust already, and now that Rust code is part of Firefox it seems unlikely that Mozilla will stop supporting it for a long time to come. It also seems to not be a zero-sum game - rather than stealing devs from C++, it's brought a lot of developers from "non-systems" languages into the "systems" space.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER54> Just as an aside how would people build something like this if they were to use say Python and try to scale to these sort of user levels? Has anyone succeeded? I'd say it would be quite a struggle without some seriously clever work!

<comment><USER55> Yeah pretty much impossible with python.

</comment></comment><comment><USER56> Hi community, Let me share my experience with you. I'm a hardcore Rails guy and I've been advocating and teaching Rails to the community for years.

My workflow for trying out a new language involves using the language for a small side project and gradually would try to scale it up. So, here's my summary, my experience of all the languages so far:

Scala - It's a vast academic language (official book is with ~700 pages) with multiple ways of doing things and it's attractiveness for me was the JVM. It's proven, robust and highly scalable. However, the language was not quite easy to understand and the frameworks that I've tried (Play 2, Lift) weren't as easy to transition to, for a Rails developer like me.

Nevertheless, I did build a simple calendar application, but it took me 2 months to learn the language and build it.

GoLang - This was my next bet, although I didn't give up on Scala completely (I know it has its uses), I wanted something simple. I used Go and had the same experience as I had when I used C++. It's a fine language, but, for a simple language, I had to fight a lot with configuration to get it working for me - (For example, it has this crazy concept of GOPATH where your project should reside and if your project isn't there it'll keep complaining). Nevertheless, I build my own (simple) Rails clone in GO and realized this isn't what I was looking for. It took my about a month to conquer the language and build my (simple) side project.

Elixir - Finally, I heard of Elixir on multiple HN Rails release threads and decided to give it a go. I started off with Phoenix. The transition was definitely wayy smoother from Rails, especially considering the founding member of this language was a Rails dev. himself (the author of "devise" gem). At first some concepts seemed different (like piping), but once I got used to it, for me there was no looking back.

All was fine until they released Phoenix 1.3, where they introduced the concept of contexts and (re) introduced Umbrella applications. Basically they encourage you to break your application into smaller applications by business function (similar to microservices) except that you can do this however you like (unopinionated). For example, I broke down my application by business units (Finance, Marketing, etc.). This forced me to re-think my application in a way I never would have thought and by this time I had finished reading all 3 popular books on this topic (Domain Driven Design). I loved how the fact that Elixir's design choices are really well suited for DDD. If you're new to DDD I suggest you try giving it a shot, it really can force you to re-think the way you develop software.

By the end of two weeks after being introduced to Elixir, I picked up the language. In a month and a half, I built a complete Salesforce clone just working on the weekends. And this includes even the UI. And I love how my application is always blazing fast, picks up errors even before it compiles and warns me if I'm no using a variable I defined somewhere.

P.S there IS a small learning curve involved if you're starting out fresh:

1) IF you're used to the Rails asset pipeline, you'll need to learn some new tools like Brunch / Webpack / etc. 2) Understand about contexts & DDD (optional) if you want to better architect your application. 3) There is no return statement in Elixir!

As a Ruby developer, here are my thoughts:

1\. So, will I be developing with Rails again? Probably yes, for simpler applications / API servers. 2\. Is Ruby dying? No. In fact, I can't wait for Ruby 3.

Some drawbacks of Elixir: 1\. Relatively new, so sometimes you'll be on your own and that's okay. 2\. Fewer libraries as compared to the Ruby eco-system. But you can easily write your own. 3\. Fewer developers, but should be fairly to onboard Ruby developers.

Cheers.

</comment><comment><USER57> When have you ever read, "How Acme scaled J2EE to 5M Concurrent Users"? I became an IT architect in 1998 at IBM, the year Sun released j2ee and IBM released Websphere. I have experienced 20 years of enterprise Java and object oriented computing, and I was thrilled when Elixir came out. I was a mainframe programmer before OO became all the rage, so I never really felt at home doing objects. Functional programming feels completely natural to me though.

What I like about this article is that they shared everything they learned with the community. Thank you for that excellent experience report.

</comment><comment><USER58> "Discord clients depend on linearizability of events"

Could this be possibly be the cause of the message reordering and dropping that I experience when I'm on a spotty connection?

</comment><comment><USER59> I realize this is off topic but how does Discord make money? I can't figure out their biz model (I'm not a gamer so I didn't even know about them).

</comment><comment><USER60> Anyone know if Phoenix/Elixir have something similar to Ruby's bettererror gem? I see Phoenix has a built-in error stack trace page which looks like a clone of bettererror but it doesn't have the real-time console inside of it.

Also, I wish they had a ORM like Sequel. These two are really what is holding me back from going full in on Elixir. Anyone can care to comment on this?

<comment><USER61> Ecto is hands down the best "ORM" I've used. At no point has it ever gotten in the way of us dynamically creating queries on the fly, just following the Ecto internals (not recommended, but totally doable) and has an extremely expressive syntax. The team behind it has been rolling out new features and are extremely responsive to requests on the mailing list. I encourage you to give it a chance.

The mental shift is that "schemas" are not objects in an ORM sense, but instead are just data, or views over data. The functions come from taking in data or a Changeset and manipulating those, versus calling a function on a class.

As far as errors, Elixir 1.4.5 has much better error messages, specifically printing the args to crashes and such, and OTP 20/ Elixir 1.5 should drastically improve Dialyzer error messages. I am not a Ruby guy, so perhaps you can say what you feel is missing from Elixir's messages and how the Ruby gem improves it. Also you can literally inspect running state on the fly with the BEAM tooling, so the need for things like debuggers goes down.

<comment><USER62> Could you explain which changes to Elixir 1.5 and OTP 20 should result in better Dialyzer error messages? I didn't find anything relevant to that in the respective changelogs.

(For context, Dialyzer is a static analysis tool detecting type errors which is part of the core Erlang distribution.)

<comment><USER61> It's kind of cryptic, but the Erlang release notes say:

      OTP-14369    Application(s): compiler, dialyzer, stdlib
                   Related Id(s): PR-1367

                   The format of debug information that is stored in BEAM
                   files (when debug_info is used) has been changed. The
                   purpose of the change is to better support other
                   BEAM-based languages such as Elixir or LFE.

                   All tools included in OTP (dialyzer, debugger, cover,
                   and so on) will handle both the new format and the
                   previous format. Tools that retrieve the debug
                   information using beam_lib:chunk(Beam, [abstract_code])
                   will continue to work with both the new and old format.
                   Tools that call beam_lib:chunk(Beam, ["Abst"]) will not
                   work with the new format.

                   For more information, see the description of debug_info
                   in the documentation for beam_lib and the description
                   of the {debug_info,{Backend,Data}} option in the
                   documentation for compile.

There aren't specific notes in the Elixir 1.5 release notes, but given that it is OTP20 compatible, I assume it would be able to leverage this, perhaps with some work by Dialyxir.

[http://erlang.org/download/otp_src_20.0-rc2.readme](http://erlang.org/download/otp_src_20.0-rc2.readme)

</comment></comment></comment></comment><comment><USER63> Compared to slack discord is a much better service for large groups . Facebook uses them for react.

<comment><USER64> I just wish they were more professional in their communications. Service went down hard this weekend and their response was a cat GIF. Hard to justify to anyone in business that they should take it seriously. A shame when their tech is such a great match for professional collaboration, and all it would take to grow it organically would be _less_ marketing copy not more.

<comment><USER65> We take outages seriously internally, but users generally don't care about the nitty-gritty. For a full, detailed postmortem on that outage you can check out our status site: [https://status.discordapp.com/incidents/ywdwttd6b0hg](https://status.discordapp.com/incidents/ywdwttd6b0hg)

</comment></comment></comment><comment><USER66> Sad to see some people taking raw and insignificant benchmarks to evaluate a language[0].

[0] [https://news.ycombinator.com/item?id=14479757](https://news.ycombinator.com/item?id=14479757)

</comment><comment><USER67> Really lovely post!

I wonder how Cloud Haskell would fare in such a scenario

</comment><comment><USER7> I so appreciate write ups that get into details of microsecond size performance gains at that scale. It's a huge help for the community.

<comment><USER61> Erlang and Elixir measure in microseconds, so it'd be them throwing away information if they did otherwise!

</comment></comment><comment><USER68> What is the business model behind Discord? They boast about being free multiple times, how do they make money? Or plan to make money?

<comment><USER69> They currently have a Nitro service that's $5/month for little features, but I don't know how far that takes them towards profitability.

</comment></comment><comment><USER70> Scale!

</comment><comment><USER70> What is Discord and Elixir?

</comment><comment><USER71> "How Discord Scaled Elixir to 5M Concurrent Users"

 _click link_

[Error 504 Gateway time-out]

only on Hacker News

</comment><comment><USER72> Unlike Discord's design team who seem to just copy all of Slack's designs and assets, the Engineering team seems to have their shit together, it is delightful to read your Elixir blogposts. Good job!

<comment><USER72> yeah go on with the downvotes, you know everyone's thinking what I said! Copying your way to success is a strategy and it's not the first time either way :)

</comment></comment><comment><USER73> Problem is that Discord sucks since it does not have a dedicated server. Sorry, move along.

<comment><USER74> That's actually why it doesn't suck for the vast majority. Not everyone wants to pay $ every month so they could have their own voice / chat server.</comment></comment>


Await and Async Explained with Diagrams and Examples - <AUTHOR> http://nikgrozev.com/2017/10/01/async-await/ <comment><AUTHOR>
      1. Makes an HTTP call, waits for it to complete, and prints the result;
      2. *Then* makes other two parallel HTTP calls;
      3. Wait for both to complete, *then* print their result.

"For such a simple example, we ended up with 2 nested then callbacks ..."

I've slightly rewritten the third point above to state my point.

Which is that I appreciate how promises reflect the actual situation. When describing the requirement, you'll use the word 'then' twice, and the code with promises ends up having two 'then's in it. It might be somewhat trivial, but I like it. It seems fitting to me that I describe "do this then that" and the code will be `doThis().then(that)`

One thing about the promise example that's somewhat unfair to the promise side, there is a double nesting of the callbacks. That's not necessary.

      rp('http://example.com/')
      .then((resp)=>{
        console.log(resp)
        return Promise.all([rp('http://example.com/'), rp('http://example.com/')])
      })
      .then(resp=>{
        console.log(resp[0])
        console.log(resp[1])
      })</comment>


Coinjob – A distributed labor marketplace for computer-based work - <AUTHOR> https://coinjob.net/ <comment><USER2> This looks suspiciously like overt advertising for another useless ICO, 50% of which is premined and 30% (make that 40% including "bonuses" & "marketing") of that is going to the "founders".

If this coin ever hits an exchange I would short it hard, there's no platform, no demo, the whitepaper is 13 pages of tosh

"Microsoft Word - CoinJob Whitepaper V1.3.docx"

"Day 1: 1ETH = 1200XCJ"

So with 200m tokens at 1200 eth each, that's $26,000,000

so they only want 780,000 for marketing and 780,000 for bonuses

And how much is going to the founders?

$3,640,000, there are four of them so skim a little marketing and bonus money and that's 1m each.

<comment><USER3> A lot of the ICO "whitepapers" are comedy gold

<comment><USER4> Just like [http://Fomocoin.org/whitepaper.pdf](http://Fomocoin.org/whitepaper.pdf)

Disclosure: I wrote this "whitepaper"

</comment></comment></comment><comment><USER5> It seems like a neat idea, but do etherium-backed ICOs have any credibility right now? There are so many problems surfacing that it's not clear why anyone _can_ take them at face value.

</comment><comment><USER3> Congrats to them for front-paging this ;)

I've heard some people say that most ICOs wouldn't stand a chance at getting VC funding. So, investors of HN, what say you of this one?

To me, the first question is why does this need its own token rather than just using Ether? I can maybe see appcoins making sense for something like Status (mobile messenger platform trying to be the crypto WeChat), but this seems too small even if it's a success to merit a separate token.

<comment><USER6> i wonder how expensive is it to frontpage something on HN.

<comment><USER3> Free, probably. It doesn't take much. I imagine this gets flagkilled pretty soon. (And just to be clear, my wink above is just b/c I thought it was kind of funny, not b/c I have anything to do with it)

<comment><USER2> Well they allocated 780,000 for marketing so clearly it's a lot more expensive than any of us could have imagined.

</comment></comment></comment></comment><comment><USER7> What's with the verify endpoint. If a hacker changed the address, couldn't they change the verify endpoint too?

<comment><USER8> Token sales should always use a .eth name as the wrapper for their deposit address. That's what FunFair did with their token sale [1]. It makes it a lot harder for a hacker or scammer to trick users into sending to the wrong address, since a user-specified identifier can be made far more mnemonic than an Ethereum account address.

[1] [https://medium.com/@weka/token-sales-done-better- funfair-d38...](https://medium.com/@weka/token-sales-done-better- funfair-d38c07a03c0b)

</comment></comment><comment><USER9> I could see this being successful by undercutting the Upwork fees alone</comment>


Lego Goes Linux  - <AUTHOR> http://www.internetnews.com/blog/skerner/lego-goes-linux.html <comment><USER2> Wow this is pretty cool. I have written some C (NXC, really [1]) programs for Mindstorms and it was super fun. I built a little rover-like car and wrote a program that would make the car play with my cat in the living room. It didn't work very well, but the idea of writing programs where the output was moving an object in the physical world was novel and very pleasing. I encourage everyone here to give it a shot.

In order to test out a new program, I had to hook the brick up to my computer via USB, then copy the program over, then run it by pressing a series of buttons on the brick. Also there wasn't a good way to get debug logs in real- time (it's hard to read the screen when it's driving around the living room). Hopefully with Linux that process will get easier. The idea of SSHing into a Mindstorms brick, running a program, and watching the debugs logs on stdout is very exciting.

1\. <http://bricxcc.sourceforge.net/nbc/>

<comment><USER3> I haven't heard of NXC! It looks like NXC is basically NQC (<http://en.wikipedia.org/wiki/Not_Quite_C>) for the current-gen Mindstorms products?

(I only got a chance to play with the new Mindstorms kit briefly in college to setup a small demo, they got about 25 educational kits as part of some grant. I just used the included software.)

I got to do FIRST Lego League in grade school with the RCX 1.5 and later the RCX 2.0

AFAIK the rules prohibited us from using NQC at the time. I believe we were limited to RoboLab [a separate product sold by LabView] and the stock Mindstorms software. (*Which I should point out is _nothing_ like the current software. It was geared towards very young users, very "flashy", and quite buggy. -- It did, however, include lots of media and examples for the robots in the instruction manual.)

That regulation was probably for the best, as my elementary self would've probably been frustrated by C-style syntax. I knew it existed only because our coach and assistant coach were both software engineers.

<comment><USER4> _Which I should point out is _nothing_ like the current software. It was geared towards very young users, very "flashy", and quite buggy._

Actually, that's been my experience with the NXT-G language (the "current software") as well.

</comment></comment></comment><comment><USER5> I wonder how many of their customers are kids, and how many are full-grown geeks like me :-)

When my daughter becomes old enough, that will be the perfect excuse to buy such devices :-)

<comment><USER6> I did! My youngest daughter (12) has an interest in robotics, so the excuse was perfect. I've played with the NXT kit as much as she has, and we're already talking about getting an EV3 kit for her birthday. (I'm excited).

<comment><USER7> That's awesome about your daughter's interest in robotics!

I'm a mentor for a robotics program that competes in FIRST competition. You should check it out and see if her schools have anything like this or if you can maybe start something yourself so other kids can learn about robotics with you and your kids.

<http://www.usfirst.org/>

<comment><USER6> Yup, familiar with the program. I actually spoke with the STEM outreach director for our middle school about a year ago. She was interested, but school policies would have made it difficult to get an FLL team up and running (she had looked into it in the past).

I may take another stab at an FRC or FTC team next year as I now know (personally) the chairman of the school board. He's an engineering prof/researcher at UIUC so I expect it to be a fairly easy sell, and as a school board member, may be able to "grease the wheels" a little.

</comment></comment></comment></comment><comment><USER8> I got to see a demo of this at CES. Very neat. I asked about downloading custom firmware to the brick, and they said it would be supported. Since is runs some version of the Linux kernel now, that should really expand the range of software which can be run. It has a USB port, so WiFi networking is now also a possibility.

Since it supports Bluetooth (presumably serial port profile), that makes communication to smartphones easy.

I'm hoping 3rd parties will make more sensors and motors for this platform. I can envision uses for small solenoids, and maybe stepper motors too.

</comment><comment><USER9> I had a course on using Lego DACTA (lego mindstorms without the mindstorms brick) when I was in fifth grade. Pure awesome.

Too bad mindstorms are ridicously expensive here in Brazil, I am yet to put my hands in one.

<comment><USER10> > Too bad mindstorms are ridicously expensive here in Brazil, I am yet to put > my hands in one.

This is a lot worse than it seems. Protectionism in Brazil goes further than Mindstorms and protectionism in the world goes further than Brazil.

What stupid (and heavy lobbied and bribed) politicians will never get is that protectionism is also cultural isolationism. By keeping good and inspiring technology off the gates they are actually limiting engineering creativity and imagination. They're actually stifling innovation and progress, not stimulating it.

Brazilian culture grew so much when it imported other people's culture and modified it to their own. Jazz was imported and became bossa-nova. Soccer was brought in by British workers in the railways and became "jogo bonito". Corbousier's modernist architecture was assimilated by Oscar Niemeyer and changed into Brazilian architecture. If culture was walled by protectionism, these developments would never happen. Similarly, technological development will be repressed by protectionism.

<comment><USER11> Aren't legit copies of video games especially difficult to find in Brazil? And when you do find them they are > $120 USD? That is just what I've heard, I wonder if it is due to this protectionism that you mention?

<comment><USER9> For games it is mostly taxes ( including protectionist taxes )

A congressman here calculated that a videogame may get taxed 273% on its original import price without shipping costs.

<comment><USER11> That number is staggering! Thank you for your input. =)

</comment></comment></comment></comment></comment><comment><USER12> I can't wait to see the LEGO hacks after this is released to the public.

</comment><comment><USER13> No, LEGO Mindstorms is not a replacement for an Arduino or a Raspberry Pi, but...</comment>


Is Doxygen Dead? - <AUTHOR>

The Doxygen news list seems moribund. Does anyone know one way or the other? Documenting minds want to know! <comment><USER2> The web site was last updated on the 19th (7 days ago) [<http://www.stack.nl/~dimitri/doxygen/>]. Seems alive to me.

<comment><AUTHOR> The web site is fine--- I'm talking about the list...</comment></comment>


JuliaAcademy Launches - <AUTHOR> https://academy.juliabox.com/ <comment><USER2> When I opened this page on Firefox, I got this message:

Improve your Experience You seem to have an incompatible browser or browser version. For improved experience we suggest that you upgrade or install Chrome

<comment><AUTHOR> Wow, I get that too. If I had noticed that before I would've submitted a different link.

<comment><USER3> We're working on fixing it on the highest priority. We were not yet expecting to be on HN!</comment></comment></comment>


New York City Mayor Bloomberg to to ban Styrofoam? - <AUTHOR> http://www.foxnews.com/politics/2013/02/14/new-york-city-mayor-bloomberg-to-announce-wants-to-ban-styrofoam/ <comment><AUTHOR> It seems it is code 6 and can only be reused</comment>


Yuzhnoye SDO completed first engine test for 3rd stage engine of CYCLONE-4 - <AUTHOR> https://twitter.com/YuzhnoyeSDO/status/1165546277645488128 <comment><AUTHOR> This was first ever engine test completed by Yuzhnoye SDO since Ukraine declares its independency on August 24, 1991.[0]

[0] [https://twitter.com/YuzhnoyeSDO/status/1165156911496531968](https://twitter.com/YuzhnoyeSDO/status/1165156911496531968)</comment>


Why the World Needs CSS Developers - <AUTHOR> https://medium.com/@elad/why-the-world-needs-css-developers-318025a6f5c1 <comment><USER2> This article is wrong about React: it does not generates unreadable CSS class names. CSS in JS libraries do.

By default, using only react + react-dom, you can achieve the exact same output as "handwritten HTML + CSS".

But CSS in JS is not bad either: it generates unique class names by using murmurhash on rules and it makes a lot of sense in a component approach (you can see it like an automatic BEM method).

Even better: The main pain point of this approach (runtime cost) will soon entirely disappear. Linaria already provides a solution, Facebook will soon roll is own (called StyleX for now). It will create a stylesheet file with atomic CSS rules (one rule per class) and replace the classNames directly at compilation time, offering deduplication, eliminates of dead code and maintenability without sacrificing performances. IMHO a huge step in CSS development.

</comment><comment><USER3> Styling, layout and Dom composition are so tightly coupled that being a CSS only developer would be impossible. Particularly in the react world where everything is JavaScript, good luck trying to write a component and pass it on to someone else styling it.

I agree that the scope of a front end developer is becoming very expansive, having to worry about accessibility concerns along with styling, performance, architecture is hard, but good teams usually cover all these things well enough combined.

</comment><comment><USER4> Not every website has to be "an app". You can build pretty functional presentational websites and blogs using CSS only for front-end. I'm going as far as replacing client side scripts (e.g. google analytics) with server side alternatives for my small projects. And with features like lazy image loading already supported by browser this really makes front-end developer life easier.

</comment><comment><USER5> This is one of my most important articles, on the issue of why I think the web industry is going in the wrong direction, hope you will like it

<comment><USER6> "CSS 3, unlike CSS 2.1, is an open standard, which means that it is continuously updated. "

No, that means it's unencumbered by licensing. It being a living document means it's continuously updated.

</comment></comment><comment><USER7> If CSS is so complex that you can only be good at it at the exclusion of other fields, maybe it's too complex ?

<comment><USER8> What do you wanna do about it?

Remove CSS?

<comment><USER7> Make it less complex. CSS 4 maybe.

</comment></comment></comment><comment><USER9> Is this the author's github? [https://github.com/elad2412](https://github.com/elad2412)

<comment><USER10> > _Is this the author 's github? > [https://github.com/elad2412](https://github.com/elad2412) _

Medium accounts are inherently linked to Twitter accounts.

The author of the article is using @elad [1] which maps 1-to-1 to Twitter [2].

In their Twitter profile there is a link pointing to their personal website [3].

Their personal website has a link to their main GitHub account [4].

GitHub account @elad2412 was created on July 01, 2012.

GitHub account @SaladFork was created on July 22, 2009.

[1] [https://medium.com/@elad](https://medium.com/@elad)

[2] [https://twitter.com/elad](https://twitter.com/elad)

[3] [https://blog.eladshahar.com/](https://blog.eladshahar.com/)

[4] [https://github.com/SaladFork](https://github.com/SaladFork)

<comment><USER5> Just to correct the last 2 links are wrong, these are my other links: [3] [https://eladsc.com](https://eladsc.com) [4] [https://github.com/elad2412](https://github.com/elad2412)

(My github isn't my main thing)</comment></comment></comment>


American Horror, Ivy League Edition - <AUTHOR> http://www.newsweek.com/american-horror-ivy-league-edition-263448 <comment><USER2> > Deresiewicz says essentially the same thing, urging students to flee the Ivy > League in favor of smaller colleges like Wesleyan, Reed and Grinnell, where > the students are more passionate and receptive...

This is terrible advice and simply not true. I went to a small, solidly ranked liberal arts school. Most of the students also drank incredibly hard; they had little regard for academics; and while it was true that you could always get in contact with a professor, the faculty was of a far lower quality than you would find at an ivy league university. Research opportunities were also non- existent.

> where the students are ... less prone to unseemly résumé-padding

It's certainly true that my school did not emphasize résumé-padding, and the end result was that none of my friends who majored in STEM related fields got into top graduate programs, and none of my friends / anyone else I knew outside of such fields achieved meaningful employment following graduation.

<comment><USER3> > It's certainly true that my school did not emphasize résumé-padding, and the > end result was that none of my friends who majored in STEM related fields > got into top graduate programs,

In my experience (currently a PhD student at a top 10 STEM program) graduate programs only care at about ability to do research. There may be a few non- research activities that demonstrate this, e.g. scoring well on Putnam, but no one scores points for activism or running a club or excelling athletically, or any of the others activities I associate with the phrase resume-padding.

<comment><USER2> This is a fair point, and what I meant was something more along the lines of "It's certainly true that my school did not emphasize résumé-padding or even resume building".

No one at my school ever discussed career development, and it was a fundamental disservice to the students. The assumption on the part of the faculty was that students would go to 1 of 3 mediocre grad programs or low tier medical programs that our students were regularly admitted to, and that furthering students' development outside of undergrad was not their concern. The only reason I ended up in a better place was because of fantastic mentoring outside of school.

To put this into context, I majored in biochemistry in undergrad, and was the only person in my department (or out of anyone else that I knew in STEM) to do summer research internships at other universities (we didn't have such a thing) / in industry while I was there. I was also the only person to push for a more independent senior research thesis rather than a cookie cutter project which was handed to me.

The university I performed research at for two summers had large, well organized career counseling and job placement efforts, including multiple large scale career fairs each year. When I returned to my school and suggested that we organize a trip to a large university career fair nearby, I was met with blank stares by the faculty and the students. When I asked people what their career plans were soon after the start of senior year, no one even had a general sense of what they wanted to do or what opportunities were out there...

<comment><USER4> Just so you know how different the other side of the coin is, I go to a top research university similar to the Ivies, and of around 50 STEM students I know, every single one did some form of summer research or STEM internship. An additional 2 semesters of research are required for most STEM majors.

</comment></comment></comment></comment><comment><USER5> _> Economics, conversely, is as popular as beer, topping all majors at Harvard, Dartmouth, Princeton and Penn. In what Deresiewicz calls "a stunning convergence," it was the top major at 26 of the nation's top 40 universities and colleges._

In a society that sets up severe and aggressive punishments for having less money than someone else, why would anyone go into any other field? Only the masochists and the selfless will be outside the world of finance soon.

<comment><USER6> Well, there will probably still be an abundant number of indie developers willing to make $40k/year. They'll move to Alaska so they can live the dream. I should have kept a list of all the recent podcasts and links. Anyway, here's one from today.

[http://www.developereconomics.com/indie-app-opportunity- gone...](http://www.developereconomics.com/indie-app-opportunity-gone/)

<comment><USER7> It always amazes me when a future where everyone can make a good living, but where there may be reduced opportunities for insanely extravagant wealth concentration, makes people scoff, but I see it often. Since technology has solved the problem of distribution - the main product that companies were invented to solve - it only seems natural for centralized companies to dissolve entirely. Aside from solving distribution (both of work and of product), everything companies provide can be provided for nearly no cost by software. And companies today have tremendous overhead, doing unwise things like maintaining offices. They do nothing but drain resources and subtract value. Physical co-location of workers is worthless thanks to technology in most every industry.

And it always seems to be the people who are most dedicated to economic ideals and capitalism who refuse to see that capitalism dictates that the current system of megacorps will be defeated by capitalism itself because they are radically inefficient when compared to swarms of freelancers coordinated by and through software. Simply because we may never see another 'Goldman Sachs' strangling the wealth out of millions and putting it into the pockets of a handful who did nothing to earn it, they ignore the fact that most people only need $72k/yr to get to a point where more money won't even make them happier.

<comment><USER6> That's less than a first year salary for a first year junior banker.

[http://www.bloomberg.com/news/2014-08-20/goldman-sachs- said-...](http://www.bloomberg.com/news/2014-08-20/goldman-sachs-said-to- raise-junior-staff-salaries-20-.html)

People with kids need to pay $1000/month for day care in the NYC area. The price for an average 3 bedroom home that's commutable to NYC has to be over $500,000.

Yes, $72k/year will go a long way in Alaska.

<comment><USER8> I think you underestimate the cost of living in Alaska. I live in Juneau and it is very expensive. Maybe not NYC or SF expensive, but in 2012 the median value of a single family home was $332,000. Food and gas are also very expensive. Things might be slightly cheaper in Anchorage or Fairbanks. You might be able to find some cheap land somewhere undesirable and build a house or a cabin, but it's going to be very expensive to get building materials shipped there.

<comment><USER9> Median household income in Alaska is $61,000/year, so there are definitely lots of Alaskans who can scrimp by on $72k.

[https://www.census.gov/hhes/www/income/data/statemedian/](https://www.census.gov/hhes/www/income/data/statemedian/)

<comment><USER10> $72k/year is above average in Alaska; it is above average in every state. If you look at the census data you cite, you can see that Alaska is #7 in terms of median household in the US. I doubt that means that Alaskans are significantly richer than average Americans, it probably means that Alaska is more expensive.

It looks like melling is using Alaska as an example of a cheap place; somewhere your dollar goes farther. If you look at the "Real Value of $100" maps you see floating around[0] you'll see it is not a particularly cheap state. It might be cheaper than New York City, but I'm not at all sure it's cheaper than New York State. In fact, New York looks like it has a median income of 53k, so plenty of New Yorkers scrimp by on $72k. Based purely on maximizing the difference between their income and their neighbors, the hypothetical indie dev should go to Mississippi.

I think looking at the cost of living in a state is a better guide than median income. This cost of living chart[2] says that the cheapest states are Mississippi, Tennessee, Kentucky, Oklahoma, and Indiana, whereas the most expensive states are Hawaii, DC, New York, Alaska (ha!), and New Jersey. Based purely on minimizing their cost of living, the hypothetical indie dev should still go to Mississippi.

I am a little surprised to see that the median income of New York is so low compared to its cost of living, so a New Yorker, on average, might be better off moving to Alaska, but not by as much as some other state, I'd wager.

My surprise got me thinking, so I did a tiny bit of excel analysis, where I normalized median wages[0] based on the value of a dollar data[1], then sorted states based on normalized median wage / cost of living[2]. Utah, Nebraska, Virginia, Iowa, and Wyoming are the top 5 with that ratio, while Hawaii, New York, DC, California and Oregon (Oregon pips Alaska here, by one rank, but my point stands, Alaska is many things, but cheap isn't one of them) are the bottom 5. I think that maps roughly with my intuition that big coastal places are sort of expensive for their average salaries and that boring, but not poor places are cheap for their salaries. I'm not sure this is relevant to the indie dev who doesn't have an employer, but for those of us who work in the flesh, I guess we should go to Utah?

[0] [https://www.census.gov/hhes/www/income/data/statemedian/](https://www.census.gov/hhes/www/income/data/statemedian/) [1] [http://taxfoundation.org/blog/real-value-100-each- state](http://taxfoundation.org/blog/real-value-100-each-state) [2] [http://www.missourieconomy.org/indicators/cost_of_living/ind...](http://www.missourieconomy.org/indicators/cost_of_living/index.stm)

</comment></comment></comment></comment></comment></comment></comment><comment><USER11> As far as access to professors; I went to a school with approximately 30k undergraduates. There were some phenomenal professors there, and if you merely showed up to one of their talks and take an interest and you immediately stood out to them.

Going by my data-point of one school, it's not particularly hard to get interaction with professors, even when nearly all the freshman classes are taught by TAs.

I got to meet and interact with some wonderful physicists, just by being interested in what they had to say. I wonder if it is easier or harder to do that at an Ivy.

<comment><USER12> Easy to do in physics at Cornell and the University of Washington. Send emails or knock on doors; it's not hard to find a professor who's interested in working with interested people.

<comment><USER13> My recollections of Cornell and UW are the same. Show up and ask questions and you might wind up working in their lab.

</comment></comment></comment><comment><USER14> Research and business benefit from pooling the best talent in one place. So what if some students get in from legacy or money. That's a price paid to have a large endowment at your disposal.

Sounds like the authors of the article and the books referenced are just jealous of Ivy League students. Students joining frats? Glad they don't do that at lesser universities.

</comment><comment><USER15> This article hits on some good points, but it conflates two separate issues:

1\. The quality of education at Ivy League schools isn't living up to the hype (and cost) of their "brand".

2\. Fraternities are a destructive force in the lives of college students.

I actually agree with both points, but they are, in fact, unrelated to each other.

</comment><comment><USER7> I imagine all of those Valedictorians moving on to Ivy League status are simply gathering the benefits of a childhood designed to make them subservient, unquestioning, and anything but self-determined. Of course they end up vomiting in frat basements. They're dumped into adulthood without an iota of experience of being a human being. How are they supposed to know what to do when no one is telling them exactly what to do with every moment of their day for the first time in their entire lives?

<comment><USER16> I am not sure why your comment is getting so much hate. Are there that many valedictorians on HN? I read so much about the dangers of performance metrics in the work place, but aren't grades the most widespread and pernicious of them all? Face it, a 4.0 does not require more than average intelligence, only the ability to quickly and precisely do exactly as you are told.

<comment><USER17> > only the ability to quickly and precisely do exactly as you are told

In my experience, my 4.0 existed because I didn't do what I'm told. If I disagreed with a teacher's answer, I would write in "(e) bsilvereagle's answer" and usually get full credit. Similarly, not once did I write a Bing- Bang-Bongo paper in high school as instructed, yet I got top grades in English because I could independently construct a worthwhile essay. I don't think I ever completed a Chemistry lab as instructed either, I would stop half way through and start tinkering around and could usually come up with a hypothesis/experiment/conclusion that was much better than what the lab set out to do.

In short, don't assume every 4.0 you see is going to do exactly as they're told.

<comment><USER16> Not sure what school you went to (or even in what country - what the heck is "Bing-Bang-Bongo"?) but you only need one teacher that doesn't like your alleged independent methods to put a fly into your perfect score. You must be very fortunate to somehow have avoided this your entire high school career. Bravo.</comment></comment></comment></comment>


Discord cracks down on hate groups, shuts down alt-right server - <AUTHOR> https://www.polygon.com/2017/8/14/16145858/discord-alt-right-server-statement <comment><USER2> That's not a good idea. It's an umbrella term that covers a large number of subgroups. Are Google, GoDaddy and Discord happy to ban all Communist related content? I'm not advocating that, and I think this pushes a much larger group to the margin than most people at these organizations think.</comment>


Lighthouse scores: how to reduce render-blocking scripts - <AUTHOR> https://fly.io/articles/lighthouse-how-to-reduce-render-blocking-scripts/ <comment><USER2> Kind of a lot of words for using Async, Defer, or IntersectionObserver. But this is a weird case of using IntersectionObserver. I'd rather use it for something like lazy-loading images[1]. They have a much higher download cost and will provide a much greater boost in performance than optimizing your JavaScript. Also, by placing all of your scripts at the bottom (save for a few at the top like Modernizr), you really don't need to worry too much about async vs defer.

[1] [https://www.machmetrics.com/speed-blog/how-to-lazy- loading-i...](https://www.machmetrics.com/speed-blog/how-to-lazy-loading- images-script-slash-page-load-time/)

<comment><USER3> It appears Google will be integrating lazy loading into the browser.[0]

In my almost 20 years of development experience, I find my todo list extremely long. And a core principle that has served me well is that if a feature is coming to the browsers for free, it's cheaper and easier to wait for the feature instead of spending real time/money building a cheap copy of it, and then having to rip it out a year later for the native feature.

"Feature patience" seems a fitting description.

I have recently used IntersectionObserver to replace scroll event functionality (modifying sticky positioned headers) and it's great for that, but I don't want to use it for lazy loading, seems like way to much work.

[0] [https://groups.google.com/a/chromium.org/forum/#!msg/blink-d...](https://groups.google.com/a/chromium.org/forum/#!msg/blink- dev/czmmZUd4Vww/1-H6j-zdAwAJ)

</comment></comment><comment><USER4> > _If this is too many words, the pretty picture below should help clear > things up._

Holy crap, I knew big G hates users but this level on condescension is off the charts!

<comment><USER5> I write the Lighthouse docs. This is not one of our docs.</comment></comment>


Cheatsheets to add to your toolbox - <AUTHOR> http://techcheatsheets.com/

<comment><USER2> I have been looking for something like this for quite some time. Talk about something people want! I've already found a few to add to my binder.</comment>


Ask HN: How to limit my web app to a particular city? - <AUTHOR> Hello HN,
I am building a web app (with Django as a backend) where users can create objects with a "where" field that is meant to inform other users viewing the object of its location. Right now, it is a simple dumb text field, so there is no "geo-validation" of the input.
For the launch, I'd like to limit the creation of objects to a certain city.  How could I do so and what do you think would be the best/simplest way?
I've thought of a) setting up city-specific subdomains (like craigslist) or b) making that field "smart" so that it would "verify" the address given (like when you type in google maps) and then filter from there to only include addresses from a particular city, but the idea was to keep it simple and that seems like it would bring more problems in (what if the address is not recognized?).
If you have any thoughts or experience with this, I'd love to hear it. Thanks. <comment><USER2> I think your input should come with a map then. Integrate google maps into your application, then save the latitude and longitude values, then before saving the object, validate the address, if its not within the city then don't save it, flash a message.

<comment><AUTHOR> Hi-- thanks for the answer, it seems a lot clearer now that you've written it down. I don't really have a need for the whole map though but if I could integrate a google maps powered input with suggestions, that would be ideal. Is that possible?</comment></comment>