Dear-GitHub: Host Github by itself as an open source project - <AUTHOR> https://github.com/dear-github/dear-github/issues/304 <comment><USER2> Why? It's free for open source, reasonable for enterprise and private use and I think you could build a better one starting from scratch.

GitHub's success is largely due to the network effect and it's entrenched status as the canonical code repository.

Besides libgit2, aka "the secret sauce", is already open source. What are you waiting for?

<comment><USER3> > libgit2, aka "the secret sauce"

I would describe GitHub's real "secret sauce" as the issue-tracking, wikis, project boards, and release management parts, that don't get represented in the repo itself.

Which is to say, if you wanted to _commoditize_ GitHub (which is basically what "open-sourcing your secret sauce" means), you'd have to create some sort of library that allowed you to treat a git repo + all those other things as one structured data-object. You would be able to use said library to both operate on all those pieces of data locally; and to sync them between different Git hosting services that all share those features.

Or, better yet, figure out a way to put all those features into git itself, so that every git repo automatically transports those pieces of data alongside itself.

<comment><USER4> Fossil SCM has Integrated Bug Tracking, Wiki, and Technotes. It's not git but is distributed. It's used for sqlite project.

<comment><USER3> That's cool. I wonder why Git hasn't copied the implementation.

<comment><USER5> They are completely different. In fossil, commit history is sacred, where git users can rebase, and cherry pick to their hearts content

<comment><USER3> I meant the implementation of these side-features as objects sitting in the repo, with appropriate client commands for creating and editing them, etc.

Git already has notes, and signed commits/signed tags, which are all those same kind of "objects that just happen to be there." So they don't need to copy Fossil's _architecture_ ; they can just copy the way that said object types interact as dependents of commits (while letting them get blown away when commits themselves do.)

</comment></comment></comment></comment></comment></comment><comment><USER6> It's kind of funny how Open Source is now a strategic initiative of many big tech companies. They open source projects for a variety of reasons: 1) to attract tech talent, 2) to "build a community" (aka get customers to become unpaid tech support), 3) to increase quality by increasing eyes on lines of code, 4) to reduce cost of maintenance by allowing the community to provide development work, 5) contributing back to existing tools allows companies to re-use technology rather than having to roll their own, 6) defeat competition by releasing a supported product for free.

The last one is a really powerful move for a software company. Normally, people will use your software if it's free, but if there's performance or other limits, it decreases the number of potential users. External open source projects then pick up the user base you might have acquired later. By open sourcing your own product, you effectively eliminate the need market for the external projects, giving you back your user base (which helps your community, potentially leads to sales, etc).

For some companies this would be cutting things very close in terms of profit margins, but Microsoft has plenty of other income that it doesn't need to worry about an occasional loss. It has way more to gain from leveraging this product in all its other tech offerings, even if they made zero money off enterprise support.

</comment><comment><USER7> _> Microsoft open sourced Xamarin after acquisition and it was good for community. _

I don't think Microsoft's previous open sourcing of Xamarin is a good indicator and may lead us to misunderstand MS's strategy. To me, it would be very out of character for MS to open source Github.

Yes, MS has released things like C# compiler (Rosalyn), Visual Studio Code (Javascript Electron app), and Xamarin to the open source community.

But, MS has not open sourced CodePlex, Visual Studio Team Foundation Server, Skype, Linkedin.

I see a difference between "programming tools" and "collaboration platforms" and Github is in the 2nd category. I see no strategic reason why MS would pay $7.5 billion for Github to just turn around and open source it.

<comment><USER8> Xamarin is an easy to use leading indicator because of Nat Friedman's involvement in both. Simply in choosing Nat Friedman as new CEO it is hard not to wonder if Microsoft at least has the intention to explore Open Source as an operating option for GitHub moving forward.

> But, MS has not open sourced CodePlex, Visual Studio Team Foundation Server, > Skype, Linkedin. [...] I see a difference between "programming tools" and > "collaboration platforms" [...]

They've open source _some_ of those "collaboration platforms". It's not a clear black & white situation, I don't think.

LinkedIn has a large open source footprint: [http://linkedin.github.io/](http://linkedin.github.io/)

More importantly, Azure has a ton of Open Source: [https://azure.github.io/](https://azure.github.io/)

A lot of what might possibly be considered "secret sauce" bits of the Azure stack are open source, and you could possibly cobble together your own mini- Azure if you needed too and for some reason it wasn't just cheaper to buy Windows Servers with Azure Stack out of the box or even to just use Azure's existing cloud.

The Azure Functions Host is the biggest example off the top of my head that a lot of people imagine would be closed source but is open source.

(Another example is I've used over the years for different reasons is Azure's "Kudu" website deployment engine.)

I don't know if there's a cut/dried point where Microsoft might currently be drawing the line between its closed source stuff and open source, but "programming tools" versus "collaboration platforms" doesn't seem to be it (even before getting into semantic arguments about the fuzzy boundary between such categories).

That said, there probably is no obvious strategic reason for Microsoft to open source GitHub at this point, and maybe all that money that was spent in the purchase are plenty more reasons not to.

But Xamarin is an interesting leading indicator, and if there is a person to put in charge of GitHub with any interest in exploring the possibility of at least open sourcing more of GitHub, even if never quite "all" of GitHub, it is probably Nat Friedman.

<comment><USER7> _> LinkedIn has a large open source footprint: [http://linkedin.github.io/](http://linkedin.github.io/)

>More importantly, Azure has a ton of Open Source: [https://azure.github.io/](https://azure.github.io/) _

Sure I get those examples of "bits and pieces" being open source. Likewise, Google open sources lots of things like Protobufs (BSD license), CityHash (MIT license), and Kubernetes (Apache license) -- but they don't open source their crown jewels of proprietary source code collaboration, the Google Cloud datacenter management stack, and of course, their latest iteration of PageRank. A lot of those Azure github repos I see are open source examples for client SDKs as opposed to building a full clone of Azure that's equivalent to RedHat's OpenStack.

Those limited examples didn't seem to be the spirit of Joel Handwell's wish. I think we can presume that he wants to download the entire Github source code, compile it, and self-host it like GitLab. I could definitely see MS open sourcing bits & pieces of Github but still not give away the entire stack.

To me, it looks like MS is taking Github in the direction of a hosted service for full Application Lifecycle Management. (For example, add more features to compete with Jira.) Github-Enterprise could possibly eventually overtake Microsoft's own Team Foundation Server as the preferred release management tool. It adds to MS portfolio of other cloud services like Office 365. Likewise, if we ask for MS to _" please open source Microsoft Excel"_, it's probably not going to happen because it's not compatible with their strategy of selling Office 365 subscriptions.

<comment><AUTHOR> >Those limited examples didn't seem to be the spirit of Joel Handwell's wish.

My wish is the same wish as the open letter dear-github ([https://github.com/dear-github/dear-github](https://github.com/dear- github/dear-github)) signers and not to host open source github in other server than github.com. Just hoping to see requested feature implemented earlier by cooperating with OSS community. I wish to still use github.com after it's open sourced. So for me, the limited partial component open source can be a starting point and the open source effort do not need to go all the way to the production backend which is distributed performance optimizing hacks usually not directly affects UX/UI.

</comment></comment></comment></comment><comment><USER9> I am going to be in the minority here, but why does HN always think all software should be free and open source? GitHub generates large revenue and provides immense value for organizations and companies. It's the same argument I see on HN about people not wanting to pay for Slack and going through insane hoops and efforts to run alternatives. Makes no sense being a technical founder. I truly believe in supporting great software with my wallet. If the organization decides to make their software open source... That's awesome and great, but starting outrage campaigns and public outcry I can't get behind. Microsoft is a public company and they have every right to monetize GitHub.

</comment><comment><USER10> I really don't see the value in open sourcing GitHub when Gitlab exists if you are looking for a replacement incase MS decides to go full tyrant - unless if you believe GitHub enterprise is an amazing project.

If GitHub wasn't built from the start for hosted deployments, it will probably be both expensive and a pain in the neck to run. GitLab probably has it's issues, but I won't be surprised if there are assumptions made in GitHub's code that assumes you are essentially running on high memory / high cpu machines to reduce latency.

<comment><USER11> This would be interesting to hear about from an insider. I suspect you're right. Having worked at a company that open sources most of their code, I've seen first hand how much better the code quality is when you know it's going out to the world. When you know it's gonna be hidden and only a handful of people will ever see it, you're much less hesitant to push up hacks :-) We open sourced something that wasn't originally going to be open, and we spent weeks going through cleaning up and refactoring things to avoid public embarrassment.

<comment><USER12> What's the business value in spending weeks of developer time refactoring things that are going open source. For some industries it's required like blockchain. For average shop seems like more expense with no profit

<comment><USER11> In our case, the business value was that it was a contractual clause with a customer. The idea was that keeping a million dollar customer and spending weeks refactoring was cheaper than losing the customer and getting sued for breach of contract. So we're probably in a similar boat as block chain like you mention.

However, I think you underestimate the value of open source. Besides being the right thing to do (respects freedom, allows developers who built it to use their code later in line with license agreement, allows others to learn by contributing or at least reading, plus many other benefits) it can be a powerful recruiting tool. There's real business value in getting good hires.

</comment></comment></comment></comment><comment><USER13> I _hate_ people using "M$" un-ironically. It wasn't funny 10 years ago when it started and it's still not funny today. How can you take that one comment seriously with that "M$" part in it.

<comment><USER6> That's cool. I hated Micro$oft for a while, for being a bunch of evil money grubbing assholes who made it harder to run the operating system I wanted, applied proprietary extensions to open standards in order to force vendor lock-in [nearly destroying the open web], and who leveraged PC manufacturer deals to tax us for software we didn't want. They literally called Open Source a cancer and lied about what the GPL specified to discourage use of the software. M$ was a monopoly, and all about the cash. It was never supposed to be funny.

Also, irony means suggesting the opposite. I don't know why anyone would use M$ ironically.

</comment></comment><comment><USER14> Like the other commenter said, if it aligns with their business interest, they will. That said, I do find it funny that essentially the "home" of most open source projects itself isn't open source. I'm not a heavy user of GitHub, as we use a self hosted instance of Gitlab at work, so my only use is browsing projects & throwing up my various side projects on it, so maybe someone can enlighten me on my next question.

Does GitHub have some proprietary tech/functionality that is valuable enough to warrant it being closed source? As far as I can tell, GitHub's most valuable assets are it's large user base and the brand itself, it's actual functionality seems to be pretty standard amongst git hosting applications (Gitlab, Gitea/Gogs, Bitbucket, etc...). However, like I said, I myself am not a heavy user of it, so there could be some feature the others are lacking that GitHub would like to keep the code closed to prevent others from replicating.

<comment><USER15> I think you're right. The only "secret sauce" I could see them having are some unique operational/architectural features of their back-end code that make sense for an entity at Github's scale. No instance of Gitea/Gitlab is even in the same ballpark wrt repo count and traffic than github.com, obviously.

</comment></comment><comment><USER16> Don't matter if you say pretty please, if it aligns with their business they will do it. They've probably thought of this move already.

<comment><USER17> There is no way I can see them open sourcing GitHub. If you have studied the history of versioning control systems (ClearCear, Perforce, TFS, Subversion, etc.) you would know how insanely powerful controlling the hosting solution is. I can see Microsoft making private repos for free and/or making GitHub Enterprise free for the first 10 users though.

<comment><USER11> It's definitely powerful, but I don't think the power of github is in the actual product. Gitlab is much better in some important ways (IMHO of course), but it's still not seen much adoption. The power of Github is in the community and the fact that it's the defacto standard place to look for code. That would likely only increase if MS open sourced the code.

</comment></comment></comment><comment><USER18> Yo dawg I heard you like GitHub, so I put GitHub on GitHub so you can contribute to GitHub using GitHub.

<comment><USER19> Replace with "Linux distribution", "compiler", or any other self-referential system [https://en.m.wikipedia.org/wiki/Self- hosting](https://en.m.wikipedia.org/wiki/Self-hosting)

</comment></comment><comment><USER20> GitLab is not open source though. They call their approach "open core" which imho is just a marketing gimmick to appear as an open source solution when in fact they are not fully open source.

<comment><USER21> I am not a big fan of GitLab-the-company. But GitLab CE is licensed as MIT. You can do with it what you wilt.

I have a hunch that your actual complaint is that _they too_ do with it what _they_ wilt, and what they wilt is not "give to you everything they add on top of GitLab for free and forever".

<comment><USER20> Yes. That's what I meant, but also I don't think that's a bad thing. Building software is hard and expensive. I think is valid to only open source certain parts of your system and reserve the right to monetize other parts and features.

</comment></comment></comment><comment><AUTHOR> Misunderstandings looks like happened regarding my intention. I'm totally not interested in running my own github instance in other server just like people who signed the open letter dear-github to plead Github to add +1 button in issue which took really long to be recognized as an issue to be implemented. I'm sure if it was open source, somebody implemented the feature and made a pull request.

My intention is to make github.com "Client Side Code" better by let OSS community join into the development. So open sourcing backend and architecture is not my interest and possibly not interest of MS too. I'm not sure if Github is made of micro services, but if it is, certain client side code could be open sourced first and only that could generate great contribution to improve it.

I'm heavy user of Gitlab and disappointed because it is no longer fully open source and some enterprise features would not be welcomed to be implemented by OSS community because it conflicts with Gitlab company revenue. In my ideal world, MS splits Github source into client and backend and open source "Github client + test backend" for community to test if client code works while they make pull request and keep production backend closed. Do you think MS would be interested in this?

<comment><USER22> I think the +1 button was a really nice feature and something they should've done a while ago, but it's hard to draw the line.

If everyone contributes their pet UI feature (e.g. custom theme, use XYZ monospace font), then there will be dozens of features that only 0.01% of users use, but it increases the overall complexity of the product and maintenance burden.

<comment><AUTHOR> +1 button could work as voting system to reduce that maintenance burden too. Semi-democratic method can be introduced to merge most voted and looks good to MS.

<comment><USER10> The only thing that sounds worse than design by committee is design by democracy. Do you know of any successful projects that are managed this way?

<comment><AUTHOR> My words were not clear, I meant and edited to "semi-democratic" way. The voting is one indicator but I also believe strong vision and strategy of leader is the key to success.

And my point was not +1 button but splitting code into open source "Client side + Test-backend" and closed sourced "Production-backend". Backend can be composed of some interface and multiple implementation such as test impl and prod impl. Actually Github could be hosted in Azure and I do not care their backend as I go github.com anyway.

</comment></comment></comment></comment></comment><comment><AUTHOR> Not sure if they will join conversation but just now I invited @satyanadella @jeffmcaffer @natfriedman @defunkt @bkeepers into the Github issue. If you want them comment, cast vote with +1

[https://github.com/dear-github/dear- github/issues/304#issuec...](https://github.com/dear-github/dear- github/issues/304#issuecomment-402011835)

</comment><comment><USER23> I think it'd be cool if they open sourced it, but w/ federation built in. So self-hosted public repo's, would call back to ms github, and show up in searches/etc. You can star/comment/etc on github.com even if it's a federated/self-hosted github.

That way the social aspect is still there, regardless of if the repo is hosted ON github.com or a remote copy.

</comment><comment><USER24> after seeing what they did with xamarin, i would not be surprised if they did. but unlike xamarin, open sourcing github wouldnt help them gain more developers on their platform and i do not see any other business benefits. i would love to see this happen though

</comment><comment><USER25> They could, but I'd argue GitHub is powerful for the community, shared user accounts between projects, and integrations. None of which you'd benefit from if you deployed your own instance.

If Microsoft open sourced GitHub, I'd still use GitHub.com, for the reasons I stated above. Heck GitLab is already OSS and I don't use that because GitHub wins via the network effect[0].

PS - I have no issue with people asking for this; I'm simply pointing out the "win" if Microsoft agreed, is lower than you'd immediately anticipate.

[0] [https://en.wikipedia.org/wiki/Network_effect](https://en.wikipedia.org/wiki/Network_effect)

</comment><comment><USER26> As much as I wish this happened I don't think it will because it'll open the doors to a lot of self-hosting eating away Github Enterprise's revenue.

</comment><comment><USER27> Pretty sure MS didn't acquire GitHub for 7.5B so they can OSS it and encourage competing, fragmented solutions with a lower barrier to entry.</comment>


Why the cops won't help you when you're getting stabbed - <AUTHOR> https://www.youtube.com/watch?v=jAfUI_hETy0 <comment><USER2> Seriously? Then what's the purpose of having a police? HN members living in the USA, do you confirm this?

<comment><USER3> Sadly, yes.

[http://nypost.com/2013/01/27/city-says-cops-had-no-duty- to-p...](http://nypost.com/2013/01/27/city-says-cops-had-no-duty-to-protect- subway-hero-who-subdued-killer/)

> [C]ity lawyers are arguing that the police had no legal duty to protect > Joseph Lozito[.]

As mentioned in the video, the NYPD won that case.

According to the NYPD at the time, the cop was a hero who tackled an armed madman: [https://twitter.com/NYPDnews/status/36601438545125376](https://twitter.com/NYPDnews/status/36601438545125376)

The worst part is, some states legally require civilians to step in and assist police officers. So, if the situation had been reversed in one of those states, and a cop were the victim, the train passengers may have been liable for inaction. As it actually happened, the inaction of the cops who hid in the engineer's compartment is considered lawful.

Community-police relations in the USA are pretty bad by many accounts. There is a large amount of civil outcry over this among Americans, including many prominent black athletes and celebrities (not sure if you've heard of Black Lives Matter, but systemic police brutality is the central topic of the movement). There are videos of police officers murdering compliant and unarmed civilians, often without consequences if there is a trial at all. Aside from that, there have been situations (search "Baltimore officer planting drugs" for one such video) where police have planted incriminating evidence only to later be implicated by their surveillance cameras.

My take is that the average American (especially those who are not of European descent) appreciates the service that police officers perform, but recognizes that a slight irritation of a police officer can lead to a very aggressive, potentially violent encounter. Bravery and congeniality are not universal virtues among American cops.

I personally believe that it's safe to say that police departments in the USA protect their own kind, even to the detriment of their ability to protect civilians. Our cops get a lot of stuff right, they have great equipment, they usually come quickly when called on, many of them truly are great public servants and protectors, but they generally go unpunished when they do wrong.

American police are accustomed to use lethal force because the self- preservation argument is very strongly accepted in American courts. "I feared for my life" is enough justification for a cop to attack an unarmed civilian. Betty Jo Shelby was acquitted in Tulsa for fearfully killing a man who had put his hands on his vehicle in submission to police. A sheriff's deputy in Tennessee tased an 81-year-old woman with garden tools in her hands because he was concerned for his own safety.

I see this case as a logical extension of the permissibility of "I feared for my life," an abdication of bravery that erodes public confidence in the police. And ultimately, the loser in this specific case is the NYPD, for lionizing a coward cop who hid while a mass murderer attacked yet another person.

<comment><USER2> Thanks for the long and useful comment.

Yes, I've heard of Black Lives Matter. I read and watch a lot of things coming from the USA ;-)

I'm shocked the police has no duty to rescue, especially in this case where they were specifically looking for the killer and he was in front of them!

[https://en.wikipedia.org/wiki/Duty_to_rescue](https://en.wikipedia.org/wiki/Duty_to_rescue)

</comment></comment></comment><comment><USER4> It's worse. If you're black, you run the risk that the cops might gleefully jump in and join the stabbing.

<comment><USER4> Ugh thread fail

</comment></comment><comment><USER5> Advertising Standards should go after the police for their "Protect and Serve" slogan.</comment>


Modern, functional Common Lisp: myths and best practices - <AUTHOR> https://ambrevar.xyz/modern-common-lisp/index.html <comment><USER2> Common Lisp is also a pleasure to use for k8s-hosted services. Just deploy your Lisp-based service to k8s, forward the sly/slime port to your local system, and continue to work on it interactively with emacs and sly/slime. It's really the most interactive development approach for k8s-hosted services. See [https://github.com/container-lisp/s2i-lisp](https://github.com/container- lisp/s2i-lisp)

<comment><USER3> How do you do the forwarding? (That's an interesting question to me by itself, but I ask because it seems like that feature is just as good outside of k8s too :))

<comment><USER4> With k8s, you just create a service of type NodePort and it assigns a free external port for that service, which is mapped back to a specific port in the Lisp container.

<comment><USER5> What sort of encryption and authentication can I expect if I do that? Can anybody who guesses the port dump code into my running Lisp instance?

<comment><USER6> This is entirely to hard to find the answer to in the kubernetes/kubectl documentation. Apparently the forward is done via socat over tls:

[https://stackoverflow.com/questions/50645059/is-kubectl- port...](https://stackoverflow.com/questions/50645059/is-kubectl-port-forward- encrypted)

And socat is mentioned in:

[https://github.com/kubernetes/kubectl/blob/master/pkg/cmd/po...](https://github.com/kubernetes/kubectl/blob/master/pkg/cmd/portforward/portforward.go)

I didn't actually manage to find out what or how socat is called - but I'm also on mobile, so it's a little convoluted to browse source code.

Anyone know where the call to socat happens, after kubectl port-forward?

</comment></comment></comment></comment></comment><comment><USER7> Thanks for writing that up. Common Lisp as a language and ecosystem is so huge that we all have our own view of what CL is and how to use it to build applications. I have been actively using CL since around 1982 for prototyping and also building applications and tools but I feel like I use a small part of what is available, mostly because I prefer to use what I am used to. I should should probably spend a little less time building things with CL and a little more time studying it.

EDIT: off topic, sorry, but I have been actively evaluating CL (using an embedded web server that starts a browser) vs. Swift (using mostly SwiftUI) for a new product I want to write. I find myself using Swift like I would CL: using Playgrounds to prototype low level code and utilities, then XCode for developing the UI. To be honest, Swift and SwiftUI is a better fit technology- wise for what I want to do, but I am so much happier when working in CL.

<comment><USER8> > Common Lisp as a language and ecosystem is so huge

This is not true IMO, when compared with the languages of today. Even if you include the numerous little details of MOP, the CL spec is nowhere near as confusing as say something like C++ 17.

<comment><USER7> You are correct, of course. Java, Python, C++, etc. also have huge ecosystems.

</comment></comment></comment><comment><USER9> I've been doing some functional style programming in Common Lisp, and I was wondering what exactly should be considered functional programming.

In particular, is object identity with EQ consistent with functional programming? Constructors do not act like functions if EQ is the equality. Or should that be more "immutable programming"?

Common Lisp, because it has EQ and object identity, cannot perform some optimizations that a truly functional language's implementation could. In particular, it cannot combine equivalent function calls, and cannot merge equivalent data (hash consing or the equivalent.)

<comment><AUTHOR> Immutability is something I'm still exploring in Common Lisp. Any pointer, anyone?

<comment><USER9> It's a matter of not modifying things once constructed. CL has hooks to help do that, although a user can always get around them.

A package I've been involved with lately is fset, which is available through quicklisp, or at

[https://github.com/slburson/fset](https://github.com/slburson/fset)

It has some interesting features, including "functional setf expansion". This would turn something like

(setf (fcar x) y)

into something equivalent to

(setf x (cons y (fcdr x)))

(where "fcar" and "fcdr" are the same as "car" and "cdr", except when they are in a setf-able place form.)

(fcar and fcdr are not in fset; I used those names just for exposition here.)

It could work with nested accessors, but only if it bottoms out in a variable.

<comment><USER10> Are you aware of any memory-leak issues with fset? I once used it in a game where I updated an fset sequence in real-time. I let it run for a while, after coming back I noticed my computer was completely frozen (out of RAM).

<comment><USER11> It's inspired by Clojure after all. The language of choice for users that don't care about performance.

<comment><USER12> It probably isn't. The initial public release of Clojure was in the fall of 2007. Scott Burson has been working on FSet since at least 2004. He himself says it's inspired by Refine.

<comment><USER11> I stand corrected. I still maintain my point about (idiomatic) Clojure being terribly slow though.

<comment><USER13> And terribly beautiful ;)

Just personal preference and enjoy programming in it. Programmers give too much importance to programming languages.

Clojure can be made to run very fast (it compiles to the same bytecode as Java after all) for the hot paths at the cost at writing less idiomatic Clojure.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER14> Assuming that the subtext here is wanting to address potential reasons for Lisp's chronic underpopularity, I think that all conversations like this miss the mark. Lisp was never failing to attract many users because people hadn't experienced sufficient evangelism about all its advanced features. It fails to attract many people because it's not a fun language to get started in.

A year or so back I picked up a copy of Land of Lisp and burned through it for pleasure reading. And I was struck by how _gross_ Lisp looks in that beginner- oriented treatment. Just this huge slog of car and cdr and let/letrec/let* and the 37 flavors of equals and the function namespace. . . and, all the while, you're being told that persevering in mastering this confusing minefield of subtleties will somehow enable you to write bug-free software. I doubt it's actually fun for most people, and the grandiose claims should beggar belief for everyone. I suppose I should count myself fortunate that I got to learn Lisp in college, where there was little attempt to make it fun, and plenty of graded assignments to keep me motivated.

Racket and Clojure are right to clean up some of the language's evolutionary history. That's a start. But even then, the treatment in beginner's guides isn't all _that_ enticing. I've also skimmed through Realm of Racket and Clojure for the Brave and True, and, while both of them work hard at being entertaining (and were fun to just sit and read), they don't really succeed at dispelling the sensation that what you're mostly doing is wrangling with the language itself.

Compare with some of the more popular Python guides. They tend to be much more dryly written, but the actual flow of the guide tends to get you pretty quickly to

        from pypi import have_fun
        have_fun()

<comment><USER15> First off, Lisp is chronically POPULAR not unpopular. (It is critically unpopular.) Secondly, Lisp is the most fun you can have programming, yeah from the start. You apparently had either a very poor teacher, or taught yourself Lisp ... same thing.

<comment><USER16> The above post didn't say unpopular, they said 'underpopular'. While that's a bit of a neologism, it just implies that it's less popular than it should be.

Also: _" Lisp is the most fun you can have programming, yeah from the start,"_ while subjective, is by most accounts, wrong. Especially when referring to Common Lisp; there could hardly be a more convoluted Lisp than Common Lisp.

<comment><USER17> I learned much of my initial Lisp programming decades ago with Macintosh Common Lisp. I found it to be incredible fun to work with a tight interactive environment on my Mac.

<comment><USER12> So did I--or, rather, I started with Coral Common Lisp, which is what it was called before Apple bought Coral Software.

I found it hugely fun--so much so that it took over my brain. I've been primarily a Lisp programmer ever since.

Coral Common Lisp offered an extremely easy way in to Lisp programming that was also tons of fun to work with, and an industrial-strength development environment for the full panoply of Mac applications. At AAAI in Detroit in 1989 I mentioned to someone that I was working in Coral Common Lisp and a passer by quipped, "Oh, the good Lisp."

It ran just fine on a 1 MB Mac Plus. You could start it up with a double- click. It launched with a Listener window and a blank Lisp source file (unless you launched it by double-clicking an existing Lisp file with some code in it.) You could save an image, copy the image file to a different machine, launch it, and you would see the same windows in the same state.

Creating a fully-functional Mac window looked like:

        (make-instance 'window)

You could populate the window with working widgets with similarly simple interactions. If you built something you wanted to keep using, you could turn it into a Mac application by doing

        (save-application "Foobar")

If you preferred constructing UI by dragging together widgets by direct manipulation, you could do that, too. Once your UI was assembled, you could tell the Lisp to save the Lisp source code needed to reconstruct it.

The built-in editor, Fred, was a species of Emacs, with the Lisp-oriented features that you would expect, but you'd never have to know that if you didn't want to, because it was also a fully-featured Tesler-style modeless editor at the same time.

The stepper, inspector, apropos, and other development niceties all had Mac graphic interfaces that made them easy to discover and tinker with. They were all written in Lisp, so you could use the Lisp reflection tools to crack them open and poke around inside to see how they worked.

Coral Common Lisp made it easy to get started using Lisp to build apps--about as easy at it can possibly be. At the same time, though, it imposed no arbitrary limits on what you could do with it. When I first met Dave Vronay he was working on the GUI for SK8, and gushed about how easy it was to write window-definition resources (WDEFs) in assembly language using CCL's LAP subsystem. He was an experienced arcade-game hacker, well versed in assembler, but Coral Common Lisp was his first fully-interactive assembler, able to define and integrate new assembly-language code while the program under development was running.

The CCL of today, Clozure Common Lisp, is still a great Lisp, but it isn't as easy or as inviting for newbies. It doesn't have the same graphical environment or the tight integration with the underlying system.

In part that's because when the Clozure Common Lisp project was created (under the name "OpenMCL"), its creators had the rights to the compiler and the Lisp runtime, but not to the Macintosh graphical environment. In part it's because modern macOS is quite a bit more complex than Macintosh System 9 and its predecessors, and development systems have a lot of additional hoops to jump through nowadays to do the kinds of things that MCL was able to do.

If a modern MCL existed, I expect a lot more people would find Lisp a lot easier and more fun to get started with. I think it's perfectly possible to create such a Lisp, but it would be a heck of a lot of work.

<comment><USER17> I used MacScheme earlier, which also was a huge amount of fun - even though I did not have the extension to access the Mac Toolbox (Toolsmith, IIRC).

MCL was a whole new world then. First with Object Lisp and later with CLOS. I used MCL before it was owned by Apple - at one time it was called MACL (from a market agreement with Franz, IIRC). But later LispWorks was another step up, because it was a __big __grown up full extended Common Lisp with everything from the commercial UNIX workstation Lisps (like Allegro CL, Lucid CL, LispWorks): it suddenly ran on small and simple to use Apple or Windows laptops, plus it had a Cocoa port.

But, as you describe, the simplicity and integration of MCL into the early MacOS was a lot of fun and there was a lot of tinkering by users. Some friends were still using MCL years after it was obsolete...

<comment><USER12> I started with the full version of MacScheme about the same time that I started with CCL. I loved them both, and at first I liked MacScheme better because Scheme was a smaller, simpler language, and easier to learn. CCL won me over in the end because it was just so much easier to get things done.

Later on, John Ulrich, who owned Lightship Software, the publisher of MacScheme, came to work at Apple and became a friend and colleague. I continued to use MacScheme off and on for years.

I've happily paid for Lispworks licenses many times now, but even Lispworks isn't a good solution for everything I want to use Lisp for. A modern version of CCL's graphical environment would still be a nicer place for newbies to start. You could even use Lispworks to build a product like that, if the license didn't prevent it. (Lispworks doesn't allow their product to be used to make Lisp development systems, which is understandable. They're a small company who don't want to put themselves out of business by enabling a customer to give their Crown Jewels away.)

<comment><USER17> yes, building a product to include development tools (compiler, ...) of LispWorks or Allegro CL would be expensive and not really attractive. I think Franz, Inc. has such a licensing agreement for Allegro CL, but their licensing model is different anyway.

In former times there was a market for add-one GUI or IDE tools. Lucid CL made money from licensing their Lisp to other companies, which then resold those with tools added on (like the SUN Common Lisp IDE). The Action! GUI designer was available for MCL. CLIM was an add-on product. The big Expert System development environments were available as add-on products to Lisp systems.

For a commercial customer something like LispWorks or Allegro CL might still be worth it - they are technically very stable platforms.

<comment><USER12> I tried to work out a way to reimplement my old programmable WYSIWIG word- processor, Alpaca, using Lispworks, but we've been unable to reach an agreement that works for me and for Lispworks. They don't want me to make something that someone could use to compete with their product, and I don't want to make a crippled version of Alpaca.

I wrote the original Alpaca in CCL, using its Cocoa interface. I used it to write a book, and a Japanese company approached me about licensing it for a product, but I fell ill before I could do anything more with it. By the time I recovered enough to look at it again, it had bitrotted. The macOS interfaces had changed out from under it.

I imagine I could fix it up and get it going again, but I'd really like to make a version of Alpaca that works on Linux and Windows as well as macOS. The obstacle is that I need a rich-text engine that can do a good job with page layout, and I don't know whether there's a suitable one, or which Lisp (that isn't Lispworks) would be best to use with it. Qt maybe? Perhaps with ECL? I guess I'd rather use CCL or SBCL. Anybody know of a text-and-page-layout engine or collection of libraries that will work well on macOS, Linux, and Windows with CCL or SBCL?

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER18> I really need to learn CL one of these days; I know Clojure reasonably well, and enough Lisp-Flavour-Erlang, Chicken Scheme, and Racket to be dangerous, but for some reason I seem to have completely avoided CL.

Based on the blog posts I've read, it seems like CL occupies the kind of space I want to be in: sort of the halfway point between theoretical and engineering. Is that a fair conclusion to draw?

<comment><USER19> It's a practical language that sometimes chooses less elegant ways to do things in favor of a complete, robust engineering experience. For instance, the fact defined functions sit in a different namespace than values creates mostly aesthetic ugliness where named functions and named values have different treatments.

        (defun f (x) (* x x))
        (setq g (compose f f))
        (g 5)

This is wrong in Common Lisp on many levels: f must be referred to as #'f, and g cannot be called as such, you must use funcall: (funcall g 5)

These might go against the sensibilities one might have had in learning a Lisp in the first place. But in practice, these don't stop you from writing solid, readable code.

It won't feel as "clean" or "academic" as Scheme, but you'll feel it easier to build large and efficient programs without pulling your hair out.

<comment><USER20> If I were doing the above a lot, I'd write compose as a macro (or better yet a compiler macro). In that case, the only thing above that needs to be changed is the setq, which would have to be written as

      (setf (fdefinition g) (compose f f))

Still different from Scheme, but some of the differences can be papered over with macros.

</comment></comment></comment><comment><USER21> On tools: note that the popular Atom editor has a great extension for interactive CL development, SLIMA, which has all the important features from Slime.

We can also use CL in Jupyter notebooks, see those good examples: [https://gist.github.com/WetHat/a49e6f2140b401a190d45d31e052a...](https://gist.github.com/WetHat/a49e6f2140b401a190d45d31e052af8f)

And Vim, Lem, cl-repl, Eclipse (not so interactive)… [https://lispcookbook.github.io/cl-cookbook/editor- support.ht...](https://lispcookbook.github.io/cl-cookbook/editor-support.html)

</comment><comment><USER22> I learned Common Lisp in high school over a decade ago and have been passively looking for a CL job that doesn't require relocation since. This has been hard (though I did land a few Clojure gigs). Is anyone aware of active Lisp shops?

</comment><comment><USER23> Amusing set of quotes about CL by famous lispers:

[http://xahlee.info/comp/Common_Lisp_quotations.html](http://xahlee.info/comp/Common_Lisp_quotations.html)

<comment><AUTHOR> A fun read with some good points, although many of them are a bit dated :p

</comment></comment><comment><USER24> Common Lisp is still the most fun language to use for me. Sadly lacking in all the currently fashionable languages is interactivity.

</comment><comment><USER25> > The myth: Common Lisp scripts and binaries are hard to distribute; binaries > are too big.

As far as I can tell, it's more like you have to pick one or the other. Distribution is easy if you don't mind distributing a 50 MB image (which is, of course, trivial for some applications and catastrophic for others). Most of the suggestions on reducing application size are essentially accounting sleight-of-hand that doesn't actually reduce the overall footprint (I'm reminded of the compressor that can "compress" any file by one byte... by moving the byte to the file name).

</comment><comment> tom-thistime This doesn't scratch my itch at all. I love Lisp but boring old Python has the libraries I need.

EDIT: Based on a comment and downvote I suppose I didn't make my point clear above. _I am saying that myths are not the only thing holding Lisp back._ (I may be wrong. I _hope_ I'm wrong. Big thank-you to people who are linking ways to get libraries in Lisp, especially Python libraries.)

I care 10x as much about libraries as about all these red herrings (EDIT: or myths, as you like) combined: " * Common Lisp does not have compile-time type checking. * Common Lisp is for imperative, object-oriented programming. * Common Lisp is too specialized, it's not for general-purpose development. * Common Lisp applications are hard to deploy. "

<comment><USER26> I'm always curious when this comes up: which libraries in particular do you have in mind? If you love Lisp and they're really absent, a short list would at least provide a data point for something that would be desirable and ambitious newcomers could cut their teeth on.

Anyway I don't think the library ecosystem is so dire... Lots of good libraries are distributed through quicklisp. It's pretty straightforward to wrap a C library. (You can even do C++ easily if you switch CL implementations to Clasp...) If you need Java libraries, you can switch CL implementations again and use ABCL. (But of course all your CL code and CL libraries still work.) Lastly, for Python there's [https://github.com/pinterface/burgled- batteries](https://github.com/pinterface/burgled-batteries) (and [https://github.com/snmsts/burgled- batteries3](https://github.com/snmsts/burgled-batteries3) for py3) that even in an incomplete state might suit your particular library needs.

Perhaps this capability isn't very compelling since Python can also access C without trouble (and Java via Jython, .NET via IronPython)? Well I guess all I have left to ask is whether you've considered there might be libraries (or features) in Lisp that would be needed that don't have equivalents in Python? What do you do then? One possible library for admittedly niche applications that came to mind was a hierarchical task planner ([https://github.com/shop- planner/shop3](https://github.com/shop-planner/shop3)) but I forgot someone did indeed make a Python library ([https://github.com/oubiwann/pyhop](https://github.com/oubiwann/pyhop)) based on an older version (SHOP1) of the background work, so depending on if you need the v3 features it might suffice.

<comment> tom-thistime Thanks for the links! I'll check them out. I like the "higher level" of Python libraries better than C or even Java, because I tend to write what most people would call prototype code. Small user base, small applications, and the really harsh constraint is development time from concept through validation.

No-doubt-unnecessary boilerplate: I understand that people have language wars, and it's possible you may have identified me as belonging to one side or another, but I'll pass. That whole state of affairs is just unfortunate.

</comment></comment></comment><comment><USER27> I hate the "lists of myths" (that aren't) genre.

> Common Lisp does not have compile-time type checking.

Nothing in the standard mentions compile time checking requirements and there is no useful de-facto standard that you (or tooling!) could seriously build upon either. I'd be suprised if python did not have better "compile-time type checking" for all practical purposes. Yes, SBCL gives much better type warnings at compile time than python, but for python you have mypy and it's A Thing, and still a joke compared to a proper type system like Ocaml's.

> Common Lisp is too specialized, it's not for general-purpose development.

Common Lisp has no eco-system to speak of for machine learning, web development, command-line utilities, games programming, mobile development, GUI programming, embedded development or pretty much anything real general purpose languages do. If you have something for which Common Lisp is a good fit, you can still be commercially successful using it because not everything requires a super rich eco-system and for some problems what Common Lisp has is in fact highly competitive. But if you want to use common lisp effectively you definitely need to pre-select the problems you want to work on accordingly to an extent that's not true of python, C, C++, Rust, javascript, Go, Java, and half a dozen other languages.

> Common Lisp applications are hard to deploy.

No server or desktop or web-browser comes with common lisp pre-installed. Building common lisp is a pain, because packaging/ASDF is terrible and you don't easily get a nice and small statically linked executable out either. Compared to exactly what language is common lisp not hard to deploy?

<comment><USER21> web development: kind of agree, but that didn't prevent me and two other colleagues to build a commercial website last year, for 1,5 month. It receives tens of thousands of visits a month and has a small admin panel for a three persons team. Job done, rent paid

CLI utils: it's simple to use cl-readline. There are a couple libraries for ncurses. The Lem editor is a good example.

mobile: nope. For the adventurous, see [https://gitlab.com/eql/EQL5-Android](https://gitlab.com/eql/EQL5-Android) and its REPL.

GUI: [https://github.com/CodyReichert/awesome- cl#gui](https://github.com/CodyReichert/awesome-cl#gui) Qt4, Gtk3, IUP, Tk, Nuklear have good bindings. Qt5 is possible with gobject-introspection (for the adventurous). Proprietary: CAPI. Electron: Ceramic. Java GUI interop with ABCL?

ML: [https://github.com/CodyReichert/awesome-cl#machine- learning](https://github.com/CodyReichert/awesome-cl#machine-learning) MGL's author won the Higgs Boson Machine Learning Challenge, but yeah.

for the rest: IDK [https://lisp-lang.org/success/](https://lisp- lang.org/success/)

<comment><USER27> CLI utils: you're missing the point. Of course you can write some simple terminal apps with common lisp, but can you name a single one that anyone who is not a lisp fanatic uses? If not, ever wonder why?

GUI: would you develop a commercial app (to pay your rent) in any of these apart from CAPI?

ML: You are joking, right?

<comment><USER21> I can name one :) pgloader [https://tapoueh.org/blog/2014/05/why-is-pgloader- so-much-fas...](https://tapoueh.org/blog/2014/05/why-is-pgloader-so-much- faster/)

I'd use Electron with Ceramic, not the GUI bindings. Or maybe Nuklear. But I would build personal projects with them.

ML: yeah, yeah. Just to show there have been work on it, that (of course) the platform is capable.

<comment><USER27> Yup, pgloader was also the one example I could think off; I was wondering if you'd name it :) Amusingly I have not been able to get the non-docker version to work due to some bogus openssl problem.

Anyway, I completely agree that Common Lisp is capable, my point also isn't that it's impossible to do things like GUIs, or write something cool and successful with it – just that the eco-system is not general-purpose language quality. If you want to make something that's successful commercially or as a well-adopted open-source project you have to be much more careful what problems you tackle than with a genuinely general purpose language.

Blog posts that claim otherwise really rub me the wrong way; I think it's possible to present CL in a favorable light without distorting reality and no one is gonna thank you later for joining the rank of people who missed out career-wise for making non-viable technology choices.

<comment><USER21> > not general-purpose language quality.

= not top-notch in all areas? Yeah, there's room for improvement. Buuut, given CL's features and stability, I will very much consider it for a commercial project in the future instead of, say, Python. I'd reject Python, for sure, for GUIs or web dev (except a small website maybe where admins would use the Django admin). Now I must find another language, and it's difficult. Many things drive me back to CL, including its state of libraries, actually pretty good compared to other languages, newer or older. No kidding.

> If you want to make something that's successful commercially or as a well- > adopted open-source project

maybe. Reading you, it seems one should avoid using CL altogether. But there's a lot of room for a successful use of CL before falling into these categories: one-off scripts, quick GUIs at work, personal tools? Commercial websites (with adequate requirements), like I did? yes, it's possible to use CL for that. Yes, it is possible to do web development for a job with CL (what kind of sites? We have to ask the few redditors that do it). My point is that CL is general-purpose enough so that it could be more used in the wild (because it is used in the wild).

Anyways. Just trying to be positive :)

<comment><USER27> > Buuut, given CL's features and stability, I will very much consider it for a > commercial project in the future instead of, say, Python[...]My point is > that CL is general-purpose enough so that it could be more used in the wild

Yeah, I'm not arguing against that at all. I just hate articles which encourage people towards doomed efforts rather than building out existing areas of strengths or address things that are real obstacles towards using them. I'd like CL to be around for at least long enough till all it's important ideas have made it into other programming languages ;)

For example, one core strength of CL is that it's the about the only really interactive/malleable/live language that can generate pretty performant code, and for sbcl you can hook deeply into the machine code generation as well. Luajit is nicer in some ways (e.g. much smaller footprint in every possible way) but CL has a much more sophisticated interactive development experience (restarts, powerful debugging, pretty-printing etc. etc.). So tooling for automation of binary rewriting, refactoring and hardening like GrammarTech seems to be working on (see [https://github.com/GrammaTech?utf8=&q=&type=&language=common...](https://github.com/GrammaTech?utf8=&q=&type=&language=common+lisp)) seems like a much more promising area to work on with common lisp to me than e.g. generic web development.

</comment></comment></comment></comment></comment></comment></comment><comment><USER28> Very nice overview, I failed to see references to LispWorks and Allegro, though.

<comment><AUTHOR> I've never used them myself, so in all honesty I have nothing to say about them! :p

That said, I support free software and I believe a proprietary compiler is not a good idea :(

<comment><USER11> Ideology aside, most proprietary compilers (mostly for Forth, Pascal and Lisp) I have used are of _excellent_ quality.

A lot also provide you the source code once you pay. AllegroCL is one of them I think.

</comment></comment></comment><comment><USER29> Lisp(Clojure) best practices: The Pure Function Pipeline Data Flow with Warehouse/Workshop Model

[https://github.com/linpengcheng/PurefunctionPipelineDataflow](https://github.com/linpengcheng/PurefunctionPipelineDataflow)

</comment><comment><USER23> I don't consider a language supportive of functional programming unless it supports tail-call optimisation, which is performed by some but not all implementations of Common Lisp.

The article recommends SBCL, which does support TCO.

An old (2011) survey of TCO support is at [https://0branch.com/notes/tco- cl.html](https://0branch.com/notes/tco-cl.html)

<comment><USER30> That sounds more like an implementation detail, in most cases. By this metric Haskell wouldn't be a functional language.

<comment><USER31> It's a _very important_ implementation detail. Explicit loops tend to imply mutation, which is contrary to idiomatic functional programming. Recursive calls don't require mutation but _do_ require TCO to achieve equivalent space complexity. Constant-factor optimizations are one thing but failing to perform TCO turns constant-space algorithms into linear-space algorithms (or linear ones into quadratic, etc.). It's less a matter of "optimizing" the calls and more a matter of not wasting limited stack space on data which is clearly not required to execute the remainder of the program. One might as well label the practice of freeing stack frames when a function returns "Function Return Optimization" (FRO) and consider it a mere "implementation detail". After all, wouldn't it be much simpler to grab new memory every time the program needs some storage space and never bother with cleaning it up? It would certainly make debugging easier with all those old variables retained for the life of the program and not constantly overwritten by new data. However, programs written for a language without guaranteed "FRO" would look very different from normal programs, much as programs designed to compensate for the lack of guaranteed TCO look very different from idiomatic functional programs.

Haskell uses a different (data-centric, non-strict) evaluation model where recursive _definitions_ don't result in recursive _calls_ , so traditional TCO isn't as relevant. Recursion is used very heavily in Haskell—which has no first-class looping constructs—but the resulting programs generally do not require large stacks. It's not unusual to be able to run even large Haskell programs with a 1 KiB maximum stack size (+RTS -K1k). Space leaks are possible, of course, but they take place in the heap.

<comment><USER17> Common Lisp was designed without requiring TCO because:

* various platforms don't support TCO. It was designed such that it can be implemented by a simple non-TCO interpreter, transpiled to a non-TCO C compiler, compiled to a non-TCO Lisp Machine CPU, or to a non-TCO virtual machine (like the JVM). Many languages don't support TCO on the JVM and may only implement explicit tail recursion or have a compiler detecting tail recursion - which is far from supporting TCO. Thus a portable conforming Common Lisp program will run in ABCL (a full Common Lisp implementation on top of the JVM) - because it will not depend on TCO to not blow up the stack, or similar.

* another reason was that Lisp has a bunch of features with don't work that well with TCO. For example Lisp always supported various dynamic scoping constructs and made extensive use of those - something which Scheme in its core does not, but provides via libraries or language extensions. Using dynamic scoping constructs makes TCO more difficult, may require a different language design, etc.

<comment><USER31> I agree with you on the first point. There are good technical reasons why TCO can't be implemented on some platforms. However, that just punts the issue one level down the stack: These platforms should have built in support for guaranteed TCO.

As for language features like dynamic scoping, I would say that a function call which needs to be followed by some cleanup activity is not in tail position, so TCO would not apply. The cleanup code could be in tail position, however, if implemented as a function call. In Common Lisp most forms of dynamic scoping or unwinding are explicit anyway, so this shouldn't come as a surprise as it might in languages like C++ and Rust where destructors are called implicitly when objects go out of scope.

<comment><USER17> They are often explicit, but they are widely used and often generated behind the scenes by macros or declarations.

<comment><USER31> Macros would need to specify whether any expressions will be evaluated in tail position. However, expressions in macros don't _look_ like they should be subject to TCO, so the default assumption should be that they aren't unless declared otherwise. Do you have any examples of cases that would be likely to cause confusion—in particular where a function call appears to occur in tail position in the code but can't be TCO'd because of a macro?

<comment><USER17> For example I see sometimes macros which generate code with compilation quality (speed, ...) declarations for all or parts of their code. Depending on the combination of qualities TCO might be enabled or disabled in code sections.

<comment><USER31> If TCO is guaranteed at the language level, as in Scheme, then it will always be enabled regardless of compilation settings. Debug builds are no more tolerant of stack space leaks than release builds. The fact that TCO isn't guaranteed is the problem here.

<comment><USER17> > If TCO is guaranteed at the language level, as in Scheme, then it will > always be enabled regardless of compilation settings

[https://www.gnu.org/software/kawa/Compatibility.html](https://www.gnu.org/software/kawa/Compatibility.html)

<comment><USER31> Your point? The page you linked to specifically says that Kawa only implements a subset of modern Scheme—by which I mean R5RS or later. Early versions of the Scheme language standard didn't require TCO, but all the recent ones do. This doesn't affect the core point that if TCO is guaranteed by the language standard, as in _modern_ Scheme, then it cannot be selectively disabled because doing so would break perfectly compliant code.</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment>


What is WebAssembly? (2015) - <AUTHOR> https://medium.com/javascript-scene/what-is-webassembly-the-dawn-of-a-new-era-61256ec5a8f6 <comment><USER2> What's the difference between asm.js and WebAssembly?

"The initial implementation of WebAssembly support in browsers will be based on asm.js and PNaCl". PNaCl? [https://en.wikipedia.org/wiki/WebAssembly](https://en.wikipedia.org/wiki/WebAssembly) , [https://en.wikipedia.org/wiki/Asm.js](https://en.wikipedia.org/wiki/Asm.js)

So is WebAssembly mere an asm.js v2 where all browser vendors agreed on a standard?

<comment><USER3> More specifically, asm.js can be seen as WebAssembly alpha, and will exist as a shiv until most browsers can run WebAssembly. Both WebAssembly and asm.js can be compiled through emscripten.

</comment></comment><comment><USER4> Note, this is from last June. It was submitted a few times, but apparently only one had a comment: [https://news.ycombinator.com/item?id=9742556](https://news.ycombinator.com/item?id=9742556)

</comment><comment><USER5> I just want a web framework that works like QT or GTK, or even a framework as a desktop that can be styled/themed independently of function.

WebAssembly should give us opportunity for this.

<comment><USER6> No, that's not really what it's about. Perhaps you're think of something more like NW.js or Electron.

<comment><USER7> I think they mean a framework that is like QT to build web apps, not a web framework to generate native apps like QT.

</comment></comment></comment><comment><USER8> I read this article several times, however I still have't understood how JS and WASM will be live together?

If I can use C/C++ with static typing(of course if I can do DOM manipulation from my C/C++ program) why I should use JS?

<comment><USER9> Why would you ever use any language over C/C++? Different abstraction levels, type systems (or lack thereof), ecosystems of libraries, even syntaxes all contribute to a decision in choosing a language.

You might not like JS personally but there are plenty of things to like about it and many other people do.

In practical terms, browsers will support JS forever for backwards compatibility even if all new code is written in WASM.

<comment><USER8> Personally I really like JavaScript and I use it many years.

</comment></comment></comment><comment><USER10> FTR, Turbo Pascal also allowed you to write in assembler - the built-in assembly was well integrated with the language, too, doing things like referencing record offsets symbolically. If you could live with the limitations, it was pretty nice.

</comment><comment><USER11> > No built-in automatic garbage collector

This is cool, because this means people will implement their own garbage collection mechanism, which could lead to interesting innovations.

<comment><USER12> > interesting innovations

Modern computers getting slow at 20 tabs of typical news sites articles is already pretty interesting. But I can't wait to see how can we innovate this even further!

<comment><USER13> _> 20 tabs of typical news sites articles_

I'm impressed you can read 20 articles at once! (Said only half jokingly. I do the same thing, but optimizing my behavior is probably the better solution to slowness)

<comment><USER12> That reminds me of an awully old joke: "I wish I had enough disc space for 2-gigabyte swap file. — Why the hell do you need a swap file so large? — I don't. I just need the space."

</comment></comment></comment></comment><comment><USER14> I read the news back in June, but haven't followed WebAssembly since, any interesting developments lately?

<comment><USER15> The V8 team recently announced that they will be implementing WASM. [https://groups.google.com/forum/#!topic/v8-users/PInzACvS5I4](https://groups.google.com/forum/#!topic/v8-users/PInzACvS5I4)

</comment></comment><comment><USER16> >A new language: WebAssembly code defines an AST (so does JavaScript) represented in a binary format. You can author and debug in a text format so it's readable.

Heh, since every binary executable has a non-compiled source code somewhere out there, can we call them readable too?

<comment><USER17> I don't think that's what they're saying; they're saying that wasm has a somewhat-readable textual representation. Not a precompiled/decompiled program, a one-one mapping from the AST to the text format. Or something.

Think of it like how assembly is to machine code, but probably more readable.

<comment><USER18> The intent is exactly like assembly/machine code, though they haven't really nailed down the format yet. Current implementation(s?) just use s-exprs, though I don't think they want that to be the final form.

</comment></comment></comment><comment><USER19> I am still a little skeptical about this new language. It is not so hard to to write optimizable JS. And when you need a real performance boost, most of such tasks are parallelizable, so you can use low-level WebGL or WebCL.

<comment> nv-vn This isn't meant to replace the code you're writing, but rather serve as a target architecture for compilers. It may also end up speeding up your JavaScript if you compile it down, but the biggest difference will be to the current "compile-to-javascript" languages (TypeScript, Elm, Dart, etc.)

</comment></comment><comment><USER20> Sounds like it's opening up a whole new world of potential exploits.

<comment><USER18> No more than the typical bounds checking done for asm.js byte buffers. It's nothing like NaCl.

</comment></comment><comment><USER21> Another thing to block in my browser I suppose.

<comment><USER22> Why? It's functionally equivalent (at least for now) to asm.js, which is just a subset of JavaScript.

</comment></comment><comment><USER23> This sounds great, but it's going to make it so, so much easier to spy on people and hide all kinds of nastiness in web applications.

Worse is that tons of web applications that get exploited won't be understood by the people who run them. Wordpress is a total minefield right now and it gets hacked all the time. What'll happen once they have a module that compiles up web assembly output for all the other modules running to speed things up?

EDIT: wordpress the thing you install yourself, not wordpress the hosted app

<comment><USER24> First:

WebAssemly will have a readable text format for view-source functionality: [https://github.com/WebAssembly/design/blob/master/TextFormat...](https://github.com/WebAssembly/design/blob/master/TextFormat.md)

Second:

 _"...so much easier to spy on people and hide all kinds of nastiness in web applications."_

What kind of spying and nastiness? WebAssembly will not have more access to your data than the JavaScript API. It will have access to the same API as JavaScript. For example if it wants your location it will call the same HTML5 location API and with the same restrictions, with the permission popup. Or setting cookies will also call the same API. And thanks to the built-in developer tools in browsers you can check the outgoing requests to see what is sent and to where.

<comment><USER23> Readable text format at a much lower level of abstraction than javascript currently is at right now. Different is different.

It's not so much about spying on your computer and gaining extra access, it's obvious (short of implementation bugs) that you won't gain any additional privileges that way.

But what you will gain is a way to obfuscate _extremely well_ "report such and such to some webserver" in a way that's difficult to detect. For example, you can hide the entropy inside of a fairly innocent looking URL and without a lot of digging you won't know _what_ that entropy represents. It can look like just a plain jane resource request and the webserver can serve up the exact same resource no matter what the entropy is, but also record that entropy for a back-channel way of exfiltrating information from your browser.

Finally, it opens up a whole new world of compiler attack. Right now the attacks against wordpress involve writing some information into a file and making it look "weird but I don't know what it does so I'd better not touch it".

What happens when breaking into a wordpress install means that you can execute the equivalent of the untraceable compiler login exploit insertion attack? You can't perform this attack without 1) a compiler and 2) a low level target that's hard to understand. You don't even need to perform a stage 3 attack which is the most sophisticated, a stage 2 would do fine.

[https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thomp...](https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf)</comment></comment></comment>


IPhone Plants vs Zombies Earns $1M in First Nine Days - <AUTHOR> http://www.gamasutra.com/view/news/27406/iPhone_Plants_vs_Zombies_Earns_1M_in_First_Nine_Days.php

<comment><USER2> This is not surprising given that the game is made by the creators of Bejeweled. It's pretty clear that they have lots of experience in what makes a highly addictive casual game.

<comment><USER3> What is surprising is that they got so much traction so fast. Does anybody know what tactics they used? Or do they have a buddy in the App Store?

<comment><USER4> Plants vs. Zombies has been the most popular flash game on popcap.com, which is the #804 website in the US(1), for a while. iPhone gamers must have a large overlap with flash gamers.

<http://www.alexa.com/siteinfo/popcap.com>

</comment></comment></comment><comment><USER5> My seven year old daughter adores this game. I'm glad to see they've found success with a game that all ages can enjoy.

</comment><comment><USER6> I believe I have all of popcap's games that are available on the iPhone, and I haven't played one I didn't like! They certainly deserve to be successful.

</comment><comment><USER7> I hate to poop into this thread (i dont want to make a new one), but can someone please tell me the link to a blog post the title of which is something to this effect:

"Customer Development: this shit really works" ?

thanks

</comment><comment><USER8> I don't get it. It's a nice Tower Defense clone, but a clone nonetheless. There are hundreds of them out there.

<comment><USER9> It has a cute little story and the design is simple and intuitive.

I played through the whole game in a matter of days and I still wanted more. They must have got something right that Tower Defense and it's clones missed.

<comment><USER10> Agreed, I've never been able to get into Tower Defense games but I (and my 3 year old, to the chagrin of my wife) have really enjoyed playing this one.</comment></comment></comment>


Elektra 0.8.21 and Book released - <AUTHOR> https://www.libelektra.org/news/0.8.21-release <comment><USER2> I spent a few minutes reading over the website trying to figure out _what_ this library does, and _why_ that's useful.

Seems to be:

\- abstracted way of reading a formatted configuration file into memory

\- knowledge of other application's config files on the same system

\- a notification system for listening to changes in configuration

\- some sorts of sysadmin tools to manage the above? Seems like it could do some of the job of Nix or Chef in terms of applying desires configurations.

So, it's like the Windows registry, except data truth is stored in regular config files (yaml, INI, /etc/hosts, etc...) in various places on your system.

Do I have that right?

I think the website could do a better job.</comment>


SpaceX Dragon Mission to the ISS [Live at 6:05AM EST] - <AUTHOR> http://www.spacex.com/webcast/?m=crs5

<comment><USER2> Discussion: [https://news.ycombinator.com/item?id=8843517](https://news.ycombinator.com/item?id=8843517)

</comment><comment><USER3> The mission has been postponed due to a technical issue. It could be as early as 9-Jan.</comment>


How we use HashiCorp Nomad - <AUTHOR> https://blog.cloudflare.com/how-we-use-hashicorp-nomad <comment><USER2> After trying out most of the kubernetes ecosystem in the pursuit of a declarative language to describe and provision services, Nomad was a breath of fresh air. It is so much easier to administer and the nomad job specs were exactly what I was looking for. I also noticed a lot of k8s apps encourage the use of Helm or even shell scripts to set up key pieces, which defeats the purpose if you are trying to be declarative about your deployment.

<comment><USER3> Helm charts are declarative way of deploying app(s) and their accompanying resources.

<comment><USER4> Helm, however, is objectively terrible with its yaml-based templating language and zero practical modularity.

<comment><USER5> Sometimes I even wish they could embed a JavaScript interpreter... After all, YAML is almost equivalent to JSON, which the perfect templating language for JSON is -- JavaScript tbh.

Or people have to keep inventing half baked things.

<comment><USER4> The problem isn't JSON or YAML: it's text templating serialization formats, instead of just marshalling/serializing them.

</comment></comment></comment></comment></comment><comment><USER6> Great writeup, even if I'm longing for more details about things like Unimog and their configuration management tool.

Pay close attention to the section where they described why they went with Nomad (Simple, Single-Binary, can schedule different workloads and not just docker, Integration with Consul). Nomad is so simple to run that you can run a cluster in dev mode with one command on your laptop (even MacOS native where you can try scheduling non-Docker workloads). I'd go so far as saying that it would even pay off to use Nomad as a job scheduler when you only have one machine and might have used systemd instead. You can wget the binary, write a small systemd unit for Nomad, then deploy any additional workloads with Nomad itself. By the time you have to scale to multiple machines you just add them to the cluster and don't have to rewrite your job definitions from systemd.

</comment><comment><USER7> The biggest hurdle with adopting nomad is the kubernetes ecosystem and related network effects. Things like operators only run on Kubernetes, and they're driving an entirely new paradigm of infrastructure and application management. HashiCorp is doing their best to keep up while supporting standard kube interfaces like CSI/CNI/CRI, but I don't know how they can possibly stay relevant with Kubernetes momentum.

In my opinion, HashiCorp should look at what Rancher did with K3s and offer something like that, integrated with the entire Hashi stack. The only reason most people choose nomad is the (initial) simplicity of it (which quickly goes away once you realize how "on an island" you are with solutions for ingress etc). Deliver kube with that simplicity and Integration and it's a much more compelling story than what Nomad delivers today.

<comment><USER8> This is what Nomad is though... It works without their other products and also natively integrates into them. Deploying Vault, Consul, and Nomad gives you a very nice experience.

Also with Consul 1.8 and Nomad 0.11 you'll get Consul Connect with Ingress gateways which solves some of those problems you mentioned.

<comment><USER9> Nomad uses consul as the kv store, so it doesn't work without their other products.

<comment><USER8> Please make an attempt to understand a technology before making comments like this. Nomad has _no_ requirement to store kv's, nor secrets. It's an entirely separate thing from Consul and Vault. It simply integrates with Consul, and Vault. It even runs its own raft store, so each product's backend is totally separate (Vault dropped it's reliance on other backends as of 1.4 and can run it's own raft store now).

Nomad template stanzas are simply consul-template ([https://github.com/hashicorp/consul- template#plugins](https://github.com/hashicorp/consul-template#plugins)), you could use `{{ with secret "" }}` and never touch Consul. You also have every function beyond service and the key* set of functions. You could build pretty static, or dynamic configurations using these blocks without ever touching Consul. On top of that, both Terraform templates and Levant work well for templating job specs out themselves, which contain template stanzas in them.

An example of something helpful would be if you wanted to drop a very small binary that changes with each deploy, you could use [https://github.com/hashicorp/consul- template#base64decode](https://github.com/hashicorp/consul- template#base64decode) and just change the contents each time you deploy the job.

If you wanted to use redis keys in your consul-template, simply drop a binary on each server as a plugin to consul-template, then for example: `{{ "my-key" | plugin "redis" "get" }}`.

Why not try running it yourself?

`nomad agent -dev` will get you a server running, then `nomad job init` will give you a full spec which you can run with `nomad job run example.job`.

<comment><USER9> I understand nomad, and what you're saying is not how anyone runs it. Directly from their documentation:

"Nomad schedules workloads of various types across a cluster of generic hosts. Because of this, placement is not known in advance and you will need to use service discovery to connect tasks to other services deployed across your cluster. Nomad integrates with Consul to provide service discovery and monitoring."

So one of the very basic features of an orchestrator, which is service discovery, requires consul. Sure, I can use it without consul to just start jobs that don't communicate, but obviously that's that the normal use case of it, and you can see that by looking at their issues list.

</comment></comment></comment></comment></comment><comment><USER10> So basically, when CloudFlare was making the decision to adopt Nomad, they had already adopted Consul and had already built in-house a custom scheduler (unimog) for their customer traffic.

It's rather disingenuous to compare to Kubernetes-based installations by pointing out that Nomad is a single Go binary that's easy to install, because they're also running Consul and unimog. For what it's worth, kubelet (which schedules jobs, like Nomad) is also a single Go binary that's easy to install, as is kube-proxy (which helps services running on a node to send traffic to the right node, so, roughly analogous to Consul).

If anything, the author practically makes the case against Nomad. Practically speaking, the only reason why CloudFlare adopted Nomad is because the stars aligned on their stack. Most companies will actually reduce complexity by adopting Kubernetes instead, compared to running two different schedulers for jobs and externally-managed service discovery.

<comment><USER9> The single binary thing is a tired trope. Kubernetes has had hyperkube for many years, which is a single binary for many years.

But in really, a single binary should be at the bottom of your list of concerns.

<comment><USER11> Hyperkube is several binaries bundled in one container, no?

<comment><USER12> It's often _distributed_ as a container, but it ~~is~~ was a single monolithic go binary. However, while I was looking for supporting evidence to that claim, I discovered that it recently[1] was evicted from the github.com/k/k repo and the docker image that is now built uses a shell script named hyperkube and the Dockerfile just untars the underlying binaries into /usr/local/bin

I wasn't able to immediately track down the alleged new repo for hyperkube, though

1 = [https://github.com/kubernetes/kubernetes/pull/83454](https://github.com/kubernetes/kubernetes/pull/83454)

</comment></comment></comment></comment><comment><USER13> I'm surprised Hashicorp hasn't repositioned this product.

Terraform was a huge breath of fresh air after Cloudformation. If you ask me, deploying apps via k8s is even better.

Everyone wants a free lunch and for me, momentum + cloud vendor support + ultimately the Nomad Enterprise features that come free with K8s made the choice easy.

<comment><USER14> TF can be often suffocating still. Pulumi runs circles around it, when it comes to user experience in writing complex, modular, composable and reusable configurations.

<comment><USER15> Except now you have to deal with javascript. It's hard enough to turn some operators towards an IaC approach, but throwing Javascript at them isn't going to help.

<comment><AUTHOR> Pulumi does not require JavaScript at all - it is one of a range of options, including JavaScript, TypeScript, Python, Go, C# and F# (currently).

Disclaimer: I worked on Terraform at HashiCorp and am a contributor to Pulumi also.

<comment><USER16> Pulumi is so good (even if it is just typed-SDK's over Terraform). Only sane approach to IaC IMO, using an actual programming language with types and IDE integration.

Really dig using with it Cloud Run.

<comment><AUTHOR> It's also not typed SDKs over Terraform. The best Pulumi providers are native and offer a much richer experience than the wrapped Terraform providers (see the Kubernetes support for example). I for one look forward to those rolling out across the board!

<comment><USER16> I actually have one burning question if you don't mind answering:

Pulumi has two different k8s modules, the standard one (pulumi-kubernetes) and "kubernetes-x" which has a much nicer API that reduces boilerplate:

[https://github.com/pulumi/pulumi- kubernetesx](https://github.com/pulumi/pulumi-kubernetesx)

The second one doesn't seem to be used in the Pulumi tutorial content and has less stars/recognition.

Are these interchangeable? If so, why not use @pulumi/kubernetesx instead of the standard k8s module everywhere?

<comment><AUTHOR> They're not interchangeable - kubernetes-x builds higher level abstractions on top of the pulumi-kubernetes API which is faithful to the openAPI supplier by Kubernetes.

To my mind, the real power of having general purpose languages building the declarative model instead of a DSL is that libraries like k-x (or indeed AWS-x) can actually be built!

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER17> Do people that aren't cloudflare scale really see the need for kubernetes and/or Nomad?

Of the two Nomad seems much more sane because it does one thing only and is much simpler to manage and deploy.

That said, having have used it, we are mostly moving away from it. Consul + Docker/Docker-compose with systemd in "a service per vm" model has proved much easier to administrate to our scale (couple of datacenters, ~1k VM mark). It is actually so simple that is boring. Instead of fiddling with infrastructure the developers spend time solving business problems...

Our stack is really boring, developers can find their way around ansible and share galaxy roles to configure / provision infrastructure... Other cloud native projects like Prometheus and Fluentd give us all the visibility we need in a very straight forward and boring way.

p.s: Great article!

<comment><USER18> From my personal experience, yes, but not just for scale. I've used Kubernetes in a 300k person organization and now in a 35 person organization. Kubernetes isn't the simplest solution, but it's just reliability in a box. You can control the network, and the servers, and the load balancing all through the API which means for most things I can install a fully clustered solution with a handful of commands. Set the number of instances I want to run and forget the whole thing exists. Suddenly I'm cloud agnostic by default and the underlying K8s management is handled by the cloud provider.

In the 300k person org it was for scale and reliability. In the 35 person org it's so that 1 guy can manage everything without trying.

<comment><USER17> I see. My last work experiences have been introducing devops workflows into companies that already had a decent tech body (~200 people) and the traditional divide between IT/Dev.

I think the approach I mention ends up being a good compromise from both sides. If I were starting a company from scratch I would certainly consider it differently.

</comment></comment></comment><comment><USER8> Cloudflare, make sure you upgrade to 0.11.3, the new scheduling behavior is awesome for large clusters.

Also a massive warning to anyone wanting to use hard cpu limits and cgroups (they do work in nomad it's just not trivial), they don't work like anyone expects and need to be heavily tested.

<comment><USER19> are you serious? you've got a major company touting a technology and a key component of it is broken?

<comment><USER20> I haven't tested hard CPU quotas with Nomad but I suspect the issue mentioned above is due to cgroups/CFS and is also applicable to Kubernetes. See these slides for more details: [https://www.slideshare.net/mobile/try_except_/optimizing- kub...](https://www.slideshare.net/mobile/try_except_/optimizing-kubernetes- resource-requestslimits-for-costefficiency-and-latency-highload)

<comment> Pirate-of-SV I recall that there's been some improvements to CFS to fix this since 2018.

<comment><USER8> There's been a ton of improvements over time. That doesn't make it any less foot-gun-y, unfortunately.

I had the exact same problem with Kubernetes (and even just straight up containers), I just want to make sure folks are extremely aware of how big of a footgun it is, and to really, really, really test it well.

</comment></comment></comment></comment></comment><comment><USER21> Nice that they did a kernel patch to fix the pivot root on initramfs problem, but sadly looking at replies it won't get merged [https://lore.kernel.org/linux- fsdevel/20200305193511.28621-1...](https://lore.kernel.org/linux- firstname.lastname@example.com/)

</comment><comment><USER22> Article is missing the title's leading "How"

</comment><comment><USER23> One point i never get for companies operating their own hardware: If your problem was having a number of well known servers for the internal management services and you then move to a nomad cluster or kubernetes to schedule them dynamically, you end up with the same problem as before to schedule the well known nomad servers or kubernetes masters. So is the only advantage here that the nomad server images update less often than the images of the management services?

<comment><USER24> Not associated with CloudFlare but I've built similar stuff with Nomad.

The cost/maintenance trade-off works when you have more SPOF management hosts than Nomad servers (5). You decrease host images down to 2, Nomad server and client, versus N management images.

Though it does sound a bit like they're using config management rather than pre-build images.

Bonus, Nomad servers are more failure resistant using Raft consensus versus any N management hosts. And for discovery I found the optimal pattern is to put all of the Nomad servers in a "cluster" A record for clients to easily join (pattern works well for Consul too)

</comment></comment><comment><USER25> Off topic but... I've always wondered how Cloudflare can not charge for the bandwidth. Even the free plan is super generous (CDN, SSL, etc).

How are they making a profit when AFAIK all other CDNs charge you for the bandwidth (and I assume they have to pay for to their providers)?

<comment><USER26> See the second answer for a statement from cf [https://webmasters.stackexchange.com/questions/88659/how- can...](https://webmasters.stackexchange.com/questions/88659/how-can- cloudflare-offer-a-free-cdn-with-unlimited-bandwidth)

<comment><USER25> I'm familiar with those terms but still.

There are people with free accounts moving GBs every month through their network and I imagine those free users must account for a very large percentage of their traffic.

<comment><USER27> I'd imagine at this point they are heavily peered in most markets, driven by said free users, so there isn't a significant opex hit bandwidth -wise. Space/power opex plus network/compute hardware capex probably dominates their spend.

</comment></comment></comment></comment><comment><USER28> "... here is the CPU usage over a day in one of our data centers where each time series represents one machine and the different colors represent different generations of hardware. Unimog keeps all machines processing traffic and at roughly the same CPU utilization."

Still a mystery to me why "balancing" has SO MUCH mindshare. This is almost certainly not the optimal strategy for user experience. It is going to be much better to drain traffic away from older machines while newer machines stay fully loaded, rather than running every machine at equal utilization factor.

<comment><USER29> I'm an engineer at Cloudflare, and I work on Unimog (the system in question).

You are right that even balancing of utilization across servers with different hardware is not necessarily the optimal strategy. But keeping faster machines busy while slower machines are idle would not be better.

This is because the time to service a request is only partly determined by the time it takes while being processed on a CPU somewhere. It's also determined by the time that the request has to wait to get hold of a CPU (which can happen at many points in the processing of a request). As the utilization of a server gets higher, it becomes more likely that requests on that server will end up waiting in a queue at some point (queuing theory comes into play, so the effects are very non-linear).

Furthermore, most of the increase in server performance in the last 10 years has been due to adding more cores, and non-core improvements (e.g. cache sizes). Single thread performance has increased, but more modestly.

Putting those things together, if you have an old server that is almost idle, and a new server that is busy, then a connection to the old server will actually see better performance.

There are other factors to consider. The most important duty of Unimog is to ensure that when the demand on a data center approaches its capacity, no server becomes overloaded (i.e. its utilization goes above some threshold where response latency starts to degrade rapidly). Most of the time, our data centers have a good margin of spare capacity, and so it would be possible to avoid overloading servers without needing to balance the load evenly. But we still need to be confident that if there is a sudden burst of demand on one of our data centers, it will be balanced evenly. The easiest way to demonstrate that is to balance the load evenly long before it becomes strictly necessary. That way, if the ongoing evolution of our hardware and software stack introduces some new challenge to balancing the load evenly, it will be relatively easy to diagnose it and get it addressed.

So, even load balancing might not be the optimal strategy, but it is a good and simple one. It's the approach we use today, but we've discussed more sophisticated approaches, and at some point we might revisit this.

<comment><USER28> Thanks for the detailed reply. It would be interesting to see your plots of latency broken down by hardware class and plotted as a function of load. I'd be pretty surprised if optimal latency was achieved near idle, since in my experience latency is a U-shape with respect to utilization: bad at 100% but also bad at 0% since it takes time to wake resources that went to sleep.

I'm sure your system has its benefits, I just get triggered by "load balancing" since it is so pervasive while also being a highly misleading and defective metaphor.

</comment></comment></comment><comment><USER30> What Grafana dashboards do you all use/like for Prometheus + Nomad?

</comment><comment><USER31> Actually, I think K8S is much simpler to operate than nomad.

With nomad, you have too many options, you can run your service/job as a process or within containers, but with Kubernetes, there is only one way to do things -- container, which is really a simple choice to make.

Besides, Kubernetes got etcd builtin, so I don't have to use deploy and maintain consul.

Last but not least, I still see containers mysteriously gone, and have no idea how nomad did that. With kubernetes, such thing never happened.

</comment><comment><USER32> I love nomad and architected a large system that used it for several years. We are finally moving away from it, however, because of its lack of native support for autoscaling. I know there are third party solutions for it, but that doesn't work for us. I suspect a large number of k8s users could use nomad instead with way less overhead.

<comment><USER33> I'm curious if you tried the new autoscaler [1] that is in Tech Preview? Is there something it's missing for your scenarios?

[1] [https://www.hashicorp.com/blog/hashicorp-nomad- autoscaling-t...](https://www.hashicorp.com/blog/hashicorp-nomad-autoscaling- tech-preview/)

<comment><USER32> As you probably know, decisions like changing infrastructure architecture is a team decision and are made with the information available at the time. The nomad autoscaler just wasn't announced/released in time for us to seriously consider it. That coupled with the uncertainty around the future of Fabio made us look elsewhere. We remain staunch supporters of Hashicorp products overall. We will continue to use Terraform, Consul and a little Vault.

</comment></comment></comment><comment><USER34> Hope they allow users to access documentation for past versions of nomad. Currently there is no easy way to find if particular configuration option mentioned in the documentation is available for not.

</comment><comment><USER35> What can Nomad do that Terraform can't? I'm pretty sure you can blue-green rolling deploy services inside of Docker containers with Terraform too.

<comment><AUTHOR> They're completely different classes of tool - Nomad is a runtime component, Terraform is not.

<comment><USER35> but terraform can talk to the docker daemon and orchestrate containers.

what can you achieve with nomad that you can't achieve with terraform?

<comment><AUTHOR> \- run a non-dockerised job

\- have a restart policy outside of what the docker daemon offers

\- automatically schedule across multiple machines based on constraints including affinity and anti-affinity, bin packing etc

\- "dispatch" jobs

The Terraform Docker provider effectively provides a tiny subset of the functionality of Nomad, which provides similar functionality across many drivers, not just Docker.

</comment></comment></comment></comment><comment><USER36> I worked at a hedge fund that also used nomad. The problem, however, is not how well it scales or whatever, but the fact that all the accompanying deployment info and literature is for kubernetes, and k8s has far more features.

I like the quality of products from HashiCorp, but k8s is far, far, ahead of where nomad is.

What I _really_ want is better integration for Terraform and kubernetes. The current TF k8s support leaves much to be desired, too many things are missing or broken and I find there are several bugs that result in deployment flapping (i.e., constantly re-deploying the same thing when there are no changes).

<comment><USER37> Which features is Nomad missing? Feature count comparisons are meaningless unless the features are tied to actual important use cases. Lots of software is encrusted with rarely used features that just add complexity.

<comment><USER4> The free version of Nomad is missing, IIUC:

\- no quotas for teams/projects/organizations

\- no preemption (ie higher priority job preempts lower priority job)

\- no namespacing

So generally it's somewhat useless in organizations where there are multiple different teams that should be able to coexist on a cluster without stepping on eachothers' toes, or even where you want a CI system to access the cluster in a safe manner.

<comment><USER38> Preemption is going OSS! [https://github.com/hashicorp/nomad/blob/master/CHANGELOG.md](https://github.com/hashicorp/nomad/blob/master/CHANGELOG.md)

We fully intend to migrate more features to OSS in the future -- especially as we build out more enterprise features. As you can imagine building a sustainable business is quite the balancing act, and there's constant internal discussion.

(I'm the Nomad Team Lead at HashiCorp)

<comment><USER15> This is going to make our AI team very happy because they can just dump experiments into the cluster at low-priority so those'll be done when those are done.

It's also going to make the operators very unhappy because it'll be harder to monitor actual memory utilization (allocated memory vs memory really in use) in order to plan cluster extensions. Are there some tools around or work planned to make this kind of scaling and utilization easier?

<comment><USER38> I was going to say "yes!" because we do have some telemetry/metrics improvements coming up, but then I realized those won't address your use case. It seems like aggregating cluster resource usage by job priority is what we need for that. Our UI might be able to calculate that, but I don't think our metrics include job priority so there's no way for your existing tooling to display it.

Please file an issue or link me to an existing issue of you have time. This seems really compelling!

</comment></comment></comment></comment></comment></comment><comment><USER39> Thanks for sharing this excellent article.</comment>


No More Searching for Hashtags - <AUTHOR> https://gettrendytags.com <comment><USER2> I'm obviously not the target user, but what is the use case of this?</comment>