Re: Integrating "safe" languages into OpenBSD? (2017) - <AUTHOR> https://marc.info/?l=openbsd-misc&m=151233345723889&w=2 <comment><USER2> This is less "On Rust" and more "On accepting a rewrite of any tool into OpenBSD, on the merits of memory safety alone".

There isn't really much of a statement or judgment on Rust. At most there's an interesting point on it's value proposition:

> However there is a rampant fiction that if you supply a new safer method > everyone will use it. For gods sake, the simplest of concepts like the stack > protector took nearly 10 years for adoption, let people should switch > languages? DELUSION.

This is mostly true. Developers in particular don't generally care about security, so selling Rust as a "secure" language is not going to be enough. I've said this since 1.0. But it's not entirely true for _products_ , which often drive development - Chrome's pairing of performance and security led to tons of marketing wins.

Given that tools like "cat" etc are:

a) Not generally security sensitive

b) Target developers

I don't see anyone choosing the "rust cat" over the builtin. This is why people build tools like "bat", that _aren 't_ just memory safe copies, but they're memory safe tools that add features and target slightly different cases.

Not much else to get from this post, I think.

<comment><USER3> Yeah. Not sure about BSD, but I was wading into building the GNU coreutils and other GNU packages just yesterday. Fresh hell, they all seem to be build dependencies of each other. 'sed' is its own build dependency.

The whole C ecosystem is a joke with respect to builds—all dependencies are implicit; you're just expected to have the exact dependencies installed on your system at the exact versions and in the exact locations on the filesystem that the build tooling is willing to look. Autotools and CMake are absolutely terrible; they make Python's build ecosystem look sane and well-designed.

So no, "security" isn't the most compelling use case (for me, anyway), it's moving past these dumpster fire build systems as quickly as possible so mere mortals can build their own software. Specifically, hasten the day where my entire application dependency tree doesn't bottom out in some Autotools, CMake, shell script, project.

<comment><USER4> Agreed on a hundred thousand.

I used to feel sheepish defending npm and the whole node package ecosystem against its critics, but once I started trying to run deep learning applications or WebRTC media servers I quickly realized that some of the critics are probably coming from a much worse package management system that they've merely grown familiar with.

Want to have two pieces of software on one computer that rely on... idk, two different versions of libwebsockets? Yeah, you'll need either docker/overlayfs, or some filesystem or pkgconfig hack that I'm sure a comment reply will helpfully mention if you want two have to versions of a library installed for two different executables.

<comment> Arch-TK "two different versions of libwebsockets"

If your programs depend on different MAJOR versions of libwebsockets and libwebsockets' build system doesn't easily allow that to happen then complain to the developer of libwebsockets about it. If the developer of libwebsockets is not interested in solving this problem then complain to the developers of the programs which depend on libwebsockets that they should find a better library which doesn't suck. If they don't listen, rewrite the tools you depend on to not suck.

If, on the other hand, the programs depend on different minor versions of libwebsockets and libwebsockets broke API in such a way that just using the highest common version doesn't work then complain to the developer of libwebsockets that they should stop doing terrible things like breaking API and then complain to the developers of those applications that they shouldn't let this kind of breakage slide and should move to a different library.

If you think this is unrealistic then I think your push towards insane let's-pull-all-the-dependencies-into-one-binary/package build systems is unrealistic.

You want a sane build system? The change starts when you stop tolerating idiots who cause things not to work.

We don't tolerate engineers who build crappy bridges which break and kill people. You shouldn't be tolerating programmers who build crappy libraries which break API or don't allow two major-versions to be easily simultaneously installed and linked against.

<comment><USER4> My issue is that the default behavior is to install these libraries globally, even when they could probably be installed locally for the thing you're trying to build. (I picked libwebsockets out of the air, I don't necessarily have a complaint with that particular library)

On my old MacBook Pro I have dozens of node projects that I've downloaded over the years just to mess around. If I ever want to clean up, I can delete a project's folder, and know that everything it installed is now gone (because it would be in its node_modules folder).

On my linux desktop if I want to mess around with a cool project that's written in C/C++ I immediately pull out Docker and start trying to build up an image, since I don't want to pollute my global library folders with libraries that I might be downloading just to try out this one thing. I can then delete the docker image if I want to clean up. This works, but it feels like a workaround to the main problem of libraries being global.

<comment><USER5> The benefit of global libraries is that they are easier to update or patch. If there is a vulnerability in an npm package, you will need to update it in all your node projects (and/or rebuild all your docker images analogously).

A hybrid solution where the package manager is aware of all installed versions, can symlink to a globally installed copy and can update all symlinks would probably work.

Unfortunately, there is very little interest in fixing the problem on both sides (distro PMs and language PMs) since they both believe their own approach is right and don't care about the other group's problems.

<comment><USER6> yarn uses a global cache and symlink if I remember correctly.

and npm has an "audit" feature, which is default on, so there's quite a bit of movement in JS land to address the disadvantages of having a myriad small packages.

<comment><USER5> Its not about the size of the packages, its about managing an entire system, for whatever definition of the word "system"

In the "olden" days, you would patch the vulnerable dependency and be sure that all services running on the machine have been updated to the new version. (in theory - if everything works great and there is ABI compatibility etc etc).

Since the library is installed in a well-known system location, there are only a few paths in the filesystem that you would need to check to make sure it's up to date. This makes machine maintenance easier.

Nowadays, to patch a vulnerable dependency you need to potentially go to all individual projects, check whether they depend on it, update the version for that project, make sure to rebuild any containers that... contain it. Rinse, repeat.

There is also not a single place where the dependencies live, so you can't just run the same command across multiple machines to do the check - you have to check the directory structure of every individual service.

This can be very slow and painful.

<comment><USER6> Agreed. Yet it seems the ecosystem simply accepts this trade-off. For increased productivity, increased memory-management security, etc.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> It also doesn't help that Rust keeps on pushing on the idea that "static linking is the only way to go". This is another cargo-cult which I wish didn't end up being engrained so deep in the toolchain because while it has some merits, it also has significant drawbacks of a typical unix distribution.

Static linking might be good for folks distributing a single server binary over a fleet of machines (pretty much like Go's primary use case), or a company that only cares about a single massive binary containing the OS itself (the browser), but it stops "being cool" very quickly on a typical *nix system if you plan to have hundreds of rust tools lying around.

Size _matters_ once you have a few thousands binaries on your system (mine has short of 6k), think about the memory requirements of caching vs cold-starting each, and running those and how patching for bug or vulnerabilities will pan out. There's a reason dynamic linkage was introduced and it's _still_ worth it today.

And rebuilding isn't cheap in storage requirements either: a typical cargo package will require a few hundred megabytes of storage, likely for _each_ rebuild.

Two days ago I rebuilt the seemingly innocuous weechat-discord: the build tree takes about 850mb on disk, with a resulting binary of 30mb. By comparison, the latest emacs binary from git rebuilds in a 1/10 of the time with all the features enabled, the build tree weights 150mb (most of which are lisp sources+elc) with a results in a binary of 9mb, not stripped.

<comment><USER8> How much do your programs _really_ share? After libc, libm, and pthreads, the most common thing they link to is probably pcre, and I'm sure you can guess how many of your programs are using that.

A good linker will shave off the parts of a library you're not using, and the parts which are left over are usually not very big. The problem isn't with static linking, it's that some "developers" think that bundling an entire Chromium build with their app is a good idea.

Rust has a problem with big binaries (so does Go), but that's Rust's problem, not static linking's problem.

<comment><USER9> Dynamic linking is pretty common in GUI applications (xlib, GTK/QT). Server- side there's also libssl, xml libraries, zlib, curl.

<comment><USER10> Outside of OS distro packaging, most Qt apps actually copy dynamically linked Qt binaries to the same app folder for re-distribution to end users within some form of installer or disk image. The same probably applies to Gtk apps too, as I think the last time I manually installed Gtk for Windows for a Gtk app was a decade ago.

Dynamic linking only works if you can guarantee ABI stability and folks haven't had to deal with ABI changes since C++13 to the point where if the C++ folks can't change the ABI by C++23 we will forget it was ever a problem because we'll make the cost of change too hard. And the current C++ ABI currently makes some parts of C++ executables sub-optimal unless you hunt down your own standard-library alternatives.

Additionally, with dynamic linking, both code authors and code users now need to agree on versions to support under assumptions that newer features in newer libraries aren't worth adopting quickly. To that end, some OSes do update dynamic libraries more quickly, but doing so theoretically requires a lot more recompilation and potentially you're downloading the same binaries more than once. At that point, dynamic linking is worth less than a binary-optimized compression algorithm, no? Especially for distributing changes to said executables.

Which isn't to say, for OS distros, that dynamic linking is bad, far from it, it tends to be the only valid solution for programming against OS core components, but that in our haste to update dynamic libraries independently of code compiled for them, we tend to forget ABI compatibility and the costs to maintain API compatibility across a wide variety of dependency versions for packagers and developers (or alternatively, the lack of updates to new library features for end users).

Windows APIs never changing is the reason Windows stagnates more than macOS, where Apple is less afraid to say your older app simply won't run any longer. Linux suffers less from this, but as pointed out in the email, part of that is because POSIX implementations are relatively stable over decades, whether or not significant improvements are still possible for more modern UX or security, for example.

The details of dynamic linking on OS platforms can be found in this recent series of posts: [https://news.ycombinator.com/item?id=23059072](https://news.ycombinator.com/item?id=23059072) (in terms of stability guarantees besides libc dynamic linking)

<comment><USER11> It is also one of the reasons why macOS is not welcomed by enterprise IT for large scale deployments.

You can have dynamic linking with ABI stability with stuff like COM and UWP.

It is also the only viable way to do plugins in scenarios where IPC is too costly.

</comment></comment></comment></comment></comment><comment><USER12> Let's try to summarize what he tries to say but kinda doesn't do very well:

_Rust and OpenBSD are not a good fit._

OpenBSD keeps to certain aspects about how to do thinks which just don't work well with rust. This doesn't mean they are better or worse. _They are just different._

For example outside of rustc-dev hardly anyone compiles rust them self. As such not many work on making rust compile itself on exotic systems (and yes in 2020 i386 is a exotic system, even through it was the standard in the past). I mean improving cross compilation is for now much more important (as far as I can tell).

But OpenBSD requires the ability to self compile and support hardware for a very long time so rust ( __for now __) doesn 't fit.

Another example is that many rewrites of classical small tools provide little value to anyone _if_ they are 100% compatible. As such they tend _to not be compatible_ in favor of better interfaces/output etc. (E.g. ripgrep!!)

Lastly OpenBSD is stuck in a world where thinks move very slow. But this is also why many people use it. One the other hand rust is from a world where thinks move much faster, sure with small steps and backward compatibility but a small step every 6 weeks is still a lot over a fiew years. So they just don't fit well together.

I also believe that in the future they might fit well together at some point. But not for now and probably not in the next 3 or so years.

</comment><comment><USER13> > I wasn't implying. I was stating a fact. There has been no attempt to move > the smallest parts of the ecosystem, to provide replacements for base POSIX > utilities.

This is in fact incorrect--there is a project aiming to build all of the coreutils in Rust ([https://github.com/uutils/coreutils](https://github.com/uutils/coreutils)).

More to the point: while I do concur in the conclusion that Rust shouldn't be a part of the OpenBSD base system, the gatekeeping implied here (it's not a serious language because it's not used to build an operating system) is really toxic. Especially considering that the gate in question has already been thoroughly breached by the language in question (some universities have switched to using Rust in their OS courses, for example).

<comment><USER14> There is no gatekeeping. To actually achieve POSIX-compliance is not an easy task and requires lots of testing. If you don't do it then your replacement will break everyone's scripts. A distro maintainer will also want to retain compatibility with their supported GNU/BSD extensions so that's more work to add on.

I would still agree with his statement at least as far as BSD is concerned. Outside of Redox I haven't seen any serious projects to implement POSIX compatibility. The Rust coreutils project is a good start but from their readme it looks like they are more aiming to achieve GNU compatibility on Microsoft Windows while being MIT licensed — I don't think anyone is seriously using it to build a BSD (or even a GNU) distro. If I'm wrong about that I'd love to hear it though. Rust is a good choice to write these things in but let's be realistic about the time frame required to rewrite everything.

Plus I just downloaded those Rust coreutils and tried to build it and now I'm waiting for 400 (!!!) Rust dependencies to download and compile. Is this really appropriate for a core system component to have this many dependencies? Or is there something I'm missing here? I admit I am not familiar with best practices in Rust. As the project stabilizes I assume they will want to start eliminating the dependencies or vendoring them upstream? At what point do we decide to put these into a separate library like libbsd or gnulib?

<comment><USER15> > Or is there something I'm missing here?

There may be some platform-specific dependencies, but at least when building on Windows, I saw 67 dependencies downloaded.

One thing that may be skewing the count is that internally coreutils is packaged with one crate per command. There are about 100 commands, so you'll have seen at least that many separate crates building. They're listed here: [https://github.com/uutils/coreutils/blob/6e8c901204934029c88...](https://github.com/uutils/coreutils/blob/6e8c901204934029c883553bf8c69b9c43577714/Cargo.toml#L217)

My impression is that ecosystems where tracking dependencies is hard tend to discourage them, while ecosystems where tracking dependencies is easy encourage them. The coreutils crate depends on separate md5, sha1, sha2, and sha3 crates, if managing external dependencies was hard I imagine those would be bundled together. But it is easy, so it makes sense to organise the crates so that each just provides one specific piece of functionality. Think single responsibility principle applied to packaging.

<comment><USER16> Urgh, that's just a verification nightmare waiting to happen. Who is going to wade through all that mess, review every package, pin its version etc...? You need reproducability before you can get safety and security.

<comment><USER17> Cargo uses lockfiles by default to pin all dependencies of an application, so reproducibility shouldn't be much of an issue.

<comment><USER18> No one reads lockfiles. In python, people have successfully backdoored projects by sending a PR that upgrades a dependency and regenerates the lockfiles.

Somewhere in the 1MB lockfile patch, they pin a dependency to a version with a known remotely exploitable flaw (sometimes a zero day, sometimes not).

</comment></comment></comment></comment></comment></comment><comment><USER19> As with most things Theo says, they're aligned with being the right answer for OpenBSD's niche, which is probably not your niche.

Any new tools that have high resource needs, don't support all the more exotic/esoteric platforms that it supports, are all nonstarters if your goal is a very small core system that compiles quickly on all supported platforms.

<comment><USER20> Yes but if he had an iota of social intelligence he would say "it is an interesting project but it is not nearly ready for inclusion in OpenBSD for reasons X, Y, and Z."

Instead he personally attacks people.

</comment></comment><comment><USER21> A rewrite of core utils[1] was started in 2013.

        commit d4e96b33e343733992fad55ac840c9649cd72ede
        Author: Jordi Boggiano <email@example.com>
        Date:   Fri Aug 2 09:22:57 2013 -0700

            Initial commit

Although I hope that Redox[2] takes off more so than rust makes it into existing systems. I dream of one day ordering a System76 with Redox already installed.

[1]: [https://github.com/uutils/coreutils.git](https://github.com/uutils/coreutils.git)

[2]: [https://www.redox-os.org/](https://www.redox-os.org/)

<comment><USER22> I was going to say, all this discussion of whether you can make OS utilities in rust seems to be ignoring the entire OS that's actually being written in rust!

<comment><USER23> How long until it will have the same kind of reputation that OpenBSD has?

<comment><USER22> That's really hard to say. Creating an OS that actually sees usage is _hard_ , and involves a huge number of non-technical hurdles to adoption. It's entirely possible that Redox will technically work, but never gain any following because it doesn't solve any problem well enough to justify the switching cost.

</comment></comment></comment></comment><comment><USER24> > However there is a rampant fiction that if you supply a new safer method > everyone will use it. For gods sake, the simplest of concepts like the stack > protector took nearly 10 years for adoption, let people should switch > languages? DELUSION.

Yeah, I agree: if the sole value proposition were 'security' it would be a really slow roll. cargo is the killer app IMO. Anyone who's walked a big dependency tree - download, build, oh oops this requires libfoo - download, build, oh oops that requires libbar...

But I think lots of coders land on rust because they want or need something faster or more portable than Python or JS and rust is 'easier' than C.

<comment><USER25> Why is rust easier than C? Is it because it makes what is simple in C hard?

<comment> Ar-Curunir Because it makes what's hard in C easy: dependency management, cross- compilation, memory safety, etc.

</comment></comment></comment><comment><USER26> I have been using bat and exa along with alacritty for a while. All of them are pretty good [https://github.com/alacritty/alacritty](https://github.com/alacritty/alacritty) [https://github.com/sharkdp/bat](https://github.com/sharkdp/bat) [https://github.com/ogham/exa](https://github.com/ogham/exa)

<comment><USER27> Throw in those, + fd[0] and rg[1]

0: [https://github.com/sharkdp/fd](https://github.com/sharkdp/fd)

1: [https://github.com/BurntSushi/ripgrep](https://github.com/BurntSushi/ripgrep)

<comment><USER26> Oh cool. Thanks

</comment></comment></comment><comment><USER28> This is from 2017.

In 2020 I personally use ag 99% of the time, and when I do use grep is because its options are burned in my brain and it saves me 30 seconds by not looking up the equivalent for ag.

[https://github.com/ggreer/the_silver_searcher](https://github.com/ggreer/the_silver_searcher)

<comment><USER29> Of course, and I use ag when I'm searching source code as well.

But ag isn't a drop in replacement for grep, and when you have tooling that's depends on that, replacing it simply isn't a viable option.

</comment></comment><comment><USER30> Marc.info seems not to be responding for me. Here is an archive.org link. [https://web.archive.org/web/20190302233853/https://marc.info...](https://web.archive.org/web/20190302233853/https://marc.info/?l=openbsd- misc&m=151233345723889&w=2)

</comment><comment><USER31> > Adding such a toolchain would take make build time from 40 minutes to hours. > I don't see how that would happen.

This should have been at the top of the email because this is the main issue (along with the i386 one). As other people have noted in this thread some people have already re-written basic utilities in Rust (and other languages).

Unless someone manage to make the Rust compiler much much faster then it is currently it won't get accepted into OpenBSD anytime soon.

<comment><USER23> Yes, but are those rewrites really 100% drop-in replacements? That's a key bit here.

</comment></comment><comment><USER32> _Such ecosystems come with incredible costs. For instance, rust cannot even compile itself on i386 at present time because it exhausts the address space._

Is this true anymore or just a history note?

<comment><USER33> > For instance, rust cannot even compile itself on i386 at present time > because it exhausts the address space.

Last I checked Rust needed more than 4GB of RAM to compile itself.

<comment><USER34> Yes, this is still true. :(

</comment></comment></comment><comment><USER35> This is not "de Raadt on Rust". There isn't any de Raadt's opinion on Rust.

This is Theo de Raadt on integrating Rust utilities in OpenBSD.

</comment><comment><USER36> Interesting read and raises one big issue:

We need to make memory safe languages accessible and with that, the tools and other frameworks for a new language isn't an easy path.

Which gets to an aspect in many walks of life - if you are doing something from scratch - you can use the latest knowledge and enforce that, but knowledge moves on and bringing up the rest of the system inline with a what is needed to use that new language/knowledge is not an overnight task, it's a slow process - and even if you provide something better, you can not guarantee it will be embraced at a needed pace to keep the momentum going.

Much like anything in life, you have better trains than you did when they was invented, but legacy infrastructure precludes an instant/overnight switch, due to impact upon what is used currently.

So case of, requirements for something better like rust support, is a bit of a catch-22 as you need it there to get the people to use it so tools get written, but to get it there you kinda need the tools so that people use it to write the tools. Hence the first step is always the biggest.

Which makes you wonder how intertwined aspects of the OS and programs are, even with standards like POSIX, there is clearly a bigger larger standard needed.

This along with the dependances and you can see why docker and other application level virtualization has much going for it.

</comment><comment><USER37> > For instance, rust cannot even compile itself on i386 at present time > because it exhausts the address space.

Wow, that one was really cruel.

<comment><USER38> Seriously who cares about i386 in 2020?

<comment><USER39> \- Older computers still in use throughout the world.

\- Embedded systems and "Internet of Things" (IoT) devices

Many operating systems and toolchains continue to support i386: [https://itsfoss.com/32-bit-os-list/](https://itsfoss.com/32-bit-os-list/)

Just because you do not use it does not mean other people do not.

</comment></comment></comment><comment><USER40> (2017)

</comment><comment><USER41> Perhaps relevant: Ted Unangst recently wrote some Rust code:

[https://flak.tedunangst.com/post/reliverator](https://flak.tedunangst.com/post/reliverator)

<comment><USER42> There are actually several of the OpenBSD devs that have been writing Rust lately. Some are doing it for work, some from personal interest. Opinions vary on how much people like it.

Rust and OpenBSD share a lot of technical values though so maybe this isn't surprising. Rust values safety and correctness. OpenBSD values safety and correctness. Rust values 'Just Works' with all the nice things cargo does and the whole 'fearless refactoring / concurrency' thing. OpenBSD values 'Just Works' with sane defaults and the 'batteries included' base system.

<comment><USER23> Nothing wrong with the OpenBSD people writing in Rust, that's a step away from tossing out say 'grep' or 'ls' and replacing it with something that isn't 100% compatible. That will cause a lot of stuff to break in hard to predict ways. It's like kicking out a random foundation stone from a building and expecting no consequences.

<comment><USER42> Sorry if you interpreted my comment as advocating for tossing out grep or ls - that certainly wasn't what I was getting at. As one of the OpenBSD people that happens to quite like Rust, I'm aware of some of the other devs who have been using it, and we've talked a bit about Rust in general on occasion. The things that people appreciate about Rust tend to align with the things they like about OpenBSD, because the two projects have similar values. That's all I meant.

</comment></comment></comment></comment><comment><USER43> Working on an operating system that uses some of these modern / safer alternatives, the concerns around bootstrapping, lack of spec, lack of formal memory model, long compile times, and larger output binaries are valid. They are real and present challenges for us, that we choose to work against. It is currently extremely difficult to produce a binary of the size of bsd cat using these technologies, and very large amounts of work are needed to close that gap, if it even can be.

In a new system you may not need cat though, but if you're a unix system with a heavy shell and textual basis, you do need lightweight versions of these things - light to build, light to run, and well integrated into the environment.

What's the path out for unix systems? I have no idea, but at least as of today I have not seen a modern safe language that is ready to replace _unix_ use cases. Maybe zig? It's hello world is at least under 10kb.

</comment><comment><USER44> I want to like the rust language. The syntax is nice and the features are interesting.

My problem is the ecosystem. The packaging and build system clearly isn't designed to accommodate classic desktop applications. The FFI is really painful, and even Firefox has tons of wrapper code to just integrate with the libraries it needs to have a GUI.

In that respect I agree with Theo in that Rust developers are focusing on far too narrow of a use case, mostly web facing applications or data processing. Cargo being so similar to npm kind of confirms this. Rust isn't intended for me.

</comment><comment><USER45> Feels like I've just had a coke. Nice reply from Theo.

</comment><comment><USER46> He might just be wrong: [https://github.com/uutils/coreutils](https://github.com/uutils/coreutils)

The only reason he might still be right is if they don't aim to be POSIX- compliant. The page doesn't explicitly call out compliance as a goal, but given that it has some busybox tests, it's probably compliant or very close.

</comment><comment><USER47> Rust is good at preventing memory leaks. It doesn't matter (as much) for a command line utility that runs for one second then exits. Of course, there are now utilities written in Rust. [https://github.com/samuela/rustybox](https://github.com/samuela/rustybox)

<comment><USER48> I think Theo's reply was a bit excessive in some regards but I can also understand him being frustrated when you've been working on a very complex project for decades and somebody comes around and say "uh, why don't you rewrite it in <hype new language>?".

It's seldom a very interesting or constructive discussion. If you think Rust has something to bring to some project then you should at least take the time to write a decent proof of concept or something similar, not ask the current maintainers to rewrite their codebase in Rust/Go/Haskell/Prolog because of some purely theoretical benefits. You don't go to construction workers in the street and tell them that they should consider using different types of hammers.

<comment><USER49> I don't think Linux or bsd will ever be rewritten in rust nor should they. I do think eventually there will be an os written in rust that surpasses them in popularity, but probably on the time frame of decades.

<comment><USER50> There are not many CVEs for coreutils anyways: [https://www.cvedetails.com/product/5075/GNU- Coreutils.html?v...](https://www.cvedetails.com/product/5075/GNU- Coreutils.html?vendor_id=72)

To improve security, Rust should probably aim to replace libjpeg and other stuff which deals with malicious data a lot.

<comment><USER51> coreutils are often run on untrusted data.

</comment></comment></comment></comment></comment><comment><USER52> There's something to it, but I don't thing the future is as grim as that.

Rust is a stepping stone. It is a proof of concept that shows a lot of problems can be solved at compile time without adding huge bloat at runtime.

Yes, Rust has problems. Some of them are just because the language is new and you feel not yet fully built.

The compile times are more more insidious side effect of Rust's properties. I think, realistically, it will take some time but the problems will be resolved either in future Rust or in another language based on it. And it will be very worth the wait.

High level languages are fun and important, but it is not possible that all system code is going to be built in high level languages. As we run stuff that is more and more demanding of CPU we need a good low level language that will also tick some important checks of reliability.

</comment><comment><USER53> Interesting points. Would a rustc + cranelift solution manage to boostrap itself in a 32-bit address space?

</comment><comment><USER54> Theo is being conservative here. It may just save OpenBSD some 10 years from now when Rust is no longer in the wind due to Hooch. At that point, someone will propose rewriting coreutils in Hooch, yet it may be much more sensible to stick with C, or rather C29.

</comment><comment><USER55> BeOS/Haiku got itself into the 250 MB range of a full-featured distro, in part by emphasizing code reuse in C++. (Though my understanding is that they used GNU grep et. al.) The current macroeconomic climate is wrong for this, but what if someone funded an effort to base an implementation of an entire OS on Rust, let's say, using Linux as a bootstrap? So there would be no requirements for "base to build base," and the developers dogfooding the toolchain would be strongly motivated to make builds faster, which would be rolled back into Rust.

<comment><USER56> look into [https://www.redox-os.org/](https://www.redox-os.org/)

</comment></comment><comment><USER57> Please put a (2017) in title.

</comment><comment><USER58> This was discussed before here: [https://news.ycombinator.com/item?id=15843631](https://news.ycombinator.com/item?id=15843631)

</comment><comment><USER59> I think it would have been good to include (2017) in the title. 2-3 years is quite a long time in the area of technology, and in my view, Rust has come a long way since 2017

</comment><comment><USER60> (2017)

</comment><comment><USER61> Hm. Rust needs more than 4GB to compile itself?

<comment><USER62> Address space ≠ required RAM.

<comment><USER34> It does require more than 4GB of RAM during its own build, though.

<comment><USER62> That's simply not true. After I couldn't find any such requirement in the build instructions for Rust, I was curious and actually setup a VM with 4GB of RAM (3.8GB usable), checked out the current rust source tree and had my machine compile the whole shebang in the background.

And what can I say - after about 3 hours and max memory usage of about 31% (or a little less than 1.8GB) the compile finished without errors.

So could you please point me to the source of your claim? Did I just uncover some magic loophole that allowed me to achieve this feat while using less than 2GB of RAM? What did I do wrong/right?

<comment><USER34> We regularly hit memory exhaustion when building Rust on Fedora and openSUSE for 32-bit architectures. It's even worse for openSUSE since we wind up having to build LLVM at times too. What you're probably missing is building the debug symbols. A release build with no debug symbols will likely be fine.

See: [https://github.com/rust-lang/rust/issues/45854](https://github.com/rust- lang/rust/issues/45854)

<comment><USER62> No you don't hit memory exhaustion. You hit __address space __exhaustion. There 's a difference and it's an important one, because 64-bit architectures don't suffer from this.

Your builds don't need more memory, they need more address space. You can convince me otherwise by enabling PAE and still running into the issue. Until then I stand by my point.

<comment><USER61> You are not wrong, but note that "memory" is not just RAM, it is also the disk space that is needed for swapping pages in and out. So, the required address space you keep referring to is not some cost-less abstraction; rather, it does in fact correspond to some physical bytes - no matter where they reside at a given particular moment, and it looks like the thing does actually need more than 4G of those.

</comment></comment></comment></comment></comment></comment></comment><comment><USER63> Hmm, this was back in 2017. I'm not an OpenBSD user nor a Rust developer - does anyone know if anything's changed in the last 3 years (the memory usage compilation issues, more mature/stable Rust-based coreutils, Rust compilation times etc)?

</comment><comment><USER64> Time and time again I'm astonished how much flamewar there is on HN when it comes to discussing people's favourite languages. (Referring to the comments, not the article.)

</comment><comment><USER65> Gate keeping is a thing ... And I smell bad odors here and there from the bsd folks, but on the whole ... It's true: nobody is gonna trade across for equals on grep, ls, whatever. If rust wants to be some OS dev language it's going to have to make a better os or competive one (eg nextstep or k42 and similar efforts) ... It's going to have to go much bigger. And finally _serious_ safe code would be accompanied by frama-c / tla+ proofs .. just pointing out rust is not the high ground.

</comment><comment><USER66> > As a response to this, Theo asked rhetorically "Where's ls, where's cat, > where's grep, and ...

For my money ripgrep knocks the socks off of grep. I can't recall if it's perfectly posix compliant but it's blazing fast and has a much nicer output.

<comment><USER67> From ripgrep's FAQ:

> Do you care about POSIX compatibility? If so, then you can't use ripgrep > because it never was, isn't and never will be POSIX compatible.

</comment></comment><comment><USER68> "Grim bloaty future" wins the Internet today.

</comment><comment><USER69> Haha, Rust go brrr

</comment><comment><USER70> Re: bootstrapping the toolchain from the base system; we have mrustc now, and it can build rustc. rustc uses LLVM for codegen, and LLVM is in the base system now.

So at least that bit is there.

What I think would be a good candidate for first base-system rust component, in case you are considering taking on a task like this, is a milter for OpenSMTPD.

</comment><comment><USER25> My hero.

</comment><comment><USER71> Is this guy trying to step up to Linus's vacated throne of "angry email opensource guy"?

</comment><comment><USER72> Rust++ will solve this.

</comment><comment><USER73> So whats the point of rewriting coreutils in rust? If you're passing attacker controlled inputs into grep you're probably doing something wrong anyway.

It doesn't seem like there's much to be gained here. Network facing software is where the big wins are.

<comment><USER74> Speed. It's normally trivial to parallelize your programs.

edit: rough crowd today. Check out Send, Sync, rayon and ripgrep.

</comment></comment><comment><USER75> >> As a response to this, Theo asked rhetorically "Where's ls, where's cat, where's grep, and where's sort?"

> I wasn't implying. I was stating a fact.

It doesn't affect his point, per se, but this is a confusing way to engage the topic.

</comment><comment><USER76> > Rust cannot compile itself.

This seems worth fixing. I think this is his main point.

There is also the bigger question if and why the core utilities should be re- written in Rust. Given that safety is the main value proposition of rust it may be makes sense, but it's a much bigger discussion.

<comment><USER77> The "on i386" part that you stripped out when quoting is actually the important bit. Rust does not care about having a self-hosting 32-bit x86 toolchain, while the lack of one is a deal-breaker for OpenBSD.

</comment></comment><comment><USER78> If you can write C as well as the OpenBSD people, be my guest, keep writing C. It might take you like 20 years to get somewhere interesting.

<comment><USER79> Like OpenSSH? PF? Clang? All of the modern compilers and languages made in C?

Maybe you can create something more widespread in Rust and not being a single part of a browser.

<comment><USER11> Clang is written in C++.

The large majority of modern compilers and languages are either bootstraped or written in a mix of the language itself and C++.

<comment><USER78> And it took them forever to get anywhere. Compare LLVM with cranelift. Cranelift made so much progress very quickly.

<comment><USER11> Indeed, but lets be honest about the optimizing capabilities and supported hardware architectures of Cranelift.

</comment></comment></comment></comment></comment><comment><USER49> All the core utils have been ported to rust for the redox project, and there are actually superior alternatives in some cases available like ripgrep.

<comment><USER80> rg is great but it isn't a drop-in replacement for POSIX grep, which means even if you have rg you still need to ship another grep so all your existing scripts and programs that expect to be able to invoke POSIX grep don't break.

<comment><USER49> Yeah and there are multiple drop in replacements for grep— he was implying those alternatives didn't exist because rust wasn't capable of handling them or people weren't interesting in writing them, but I think that's been proven wrong many times over at this point.

[https://github.com/redox- os/extrautils/blob/master/src/bin/g...](https://github.com/redox- os/extrautils/blob/master/src/bin/grep.rs)

<comment><USER60> Was this true in 2017, when the statement was written?

<comment><USER81> Both redox's grep and ripgrep (which trivially demonstrates that a grep _can_ be implemented in rust even if it's not a POSIX grep implementation) were created / started in early 2016, so yes.

Not to mention Theo's statement is so much weaker than actually successful implementations the existence of these two completely blows through the goal:

> There has been _no attempt_ to move the smallest parts of the ecosystem, to > provide replacements for base POSIX utilities.

<comment><USER60> What matters is when they were feature complete, not when the project started. I can't plausibly claim that a program I wrote some code for but isn't at the state of being a viable substitute yet is a "replacement."

(And yes, Theo, like many hackers, is prone to exaggeration.)

</comment></comment></comment></comment></comment></comment><comment><USER82> > I wasn't implying. I was stating a fact. There has been no attempt to move > the smallest parts of the ecosystem, to provide replacements for base POSIX > utilities.

Huh? Some days all I see on HN fron page is "X written in Rust" projects, where X is some long existing tool (now with colorized output by default!)

</comment><comment><USER83> Every time I see a Theo de Raadt thing on the front page it's some nonsensical language rant that has the utility of a PR review comment on some random leaf JS project.

The guy does a lot of work maintaining OpenBSD but it looks like people only upvote his pointless flaming. This is practically TMZ for Nerds stuff.

I think I'll block Theo de Raadt for myself on HN.

</comment><comment><USER84> "For instance, rust cannot even compile itself on i386 at present time because it exhausts the address space."

I don't want to be harsh, but who the hell is compiling anything on i386 now? Real question, is anyone even making fresh i386 hardware that might be used by consumers, or deployed in datacentres?

<comment><USER85> OpenBSD is. Just because you can't think of a use case doesn't mean they don't exist.

<comment><USER84> I'm asking about the quality of the reason. Is there a real reason to not cross compile in 2020?

<comment><USER85> Well, they're still supporting i386 releases, which means that there are still i386 users out in the wild that aren't willing to upgrade their hardware.

</comment></comment></comment></comment><comment><USER85> Admittedly it's not much of an opinion on Rust as a language but rather the size and performance of its compiler. And I can understand why Rust advocates would see it as a criticism of the language...because the language and the compiler are heavily intertwined. I can also see that there are extremely valid use cases for small and simple over big and complex, and in those use cases, Rust is absolutely a bad fit.

I do, however, think it is odd that OpenBSD would value compiler size over security, given the philosophical values of the OpenBSD project. I would expect that sort of opinion (and agree with it) in the NetBSD project, but I would think that OpenBSD would be willing to put up with big compiler binaries and slow compile times in exchange for even small improvements in memory safety. Strongly and statically typed languages in the ML tradition (like Rust, SML, OCaml, Haskell) all have big bloaty compilers, but that is for a specific reason: the type system that prevents you from compiling invalidly- typed programs is in essence a test suite. And for rust, a language with a type system that covers memory safety bugs, that test suite is _massive_.

OpenBSD is pretty famous for the comprehensiveness of its test suite, perhaps only superseded in depth by applications that lack its breadth (such as SQLite). I've never run the full OpenBSD test suite in a development cycle, but I've heard stories about running it that perhaps parallel the frustration newbies have with getting rust programs to compile.

What would an OpenBSD test suite look like if it could conclusively prove the absence of all memory safety bugs? Would Theo De Raadt be willing to put up with a test suite that ran for two hours longer and required more memory than could be allocated on i386 systems if it could do so? I think he would... _because that is something in line with the OpenBSD philosophical values_. So why is it that it needs to compile code so efficiently, if you're gonna turn around and run tests for 3 days to make sure nothing breaks?

</comment><comment><USER86> > Yes, now I am implying something: you won't bother to rewrite the utilities.

> So we cannot replace any base utility, unless the toolchain to build it is > in the base. Adding such a toolchain would take make build time from 40 > minutes to hours. I don't see how that would happen.

So, to summarize: "We're not going to include the capability to write core utils in Rust, and also, no one is bothering to do it."

Its rather startling to me how tail-wagging-the-dog, or maybe chicken-and-egg, this situation is. Did anyone stop to consider that the reason why some of these utilities aren't being rewritten is _because_ the unix old guard doesn't want to support it in the toolchain? If this guy is the wall that you'll eventually have to crash your car into, I'm not going to get behind the wheel in the first place. Why waste my time?

Look; the compile limitations are real. I'm not saying that it _should_ be supported. But, attitudes like the one are not productive. What is it about operating system devs and these aggressive, mean, anti-social, self-important personalities? A response worded like this should not be tolerated by the community. It would have been just as easy and clear to say "We don't have toolchain support for Rust, and we're really not interested in supporting it due to its compile time and architecture support issues. <One sentence explaining compile time issues>. <One sentence explaining architecture support issues>. We can revisit this discussion in the future if/when these improve."

<comment><USER87> I got the same vibe from the email. It reads just like bigcorp turf defense.

When a smart but curmudgeonly component owner just doesn't want to accept some outsider's change, there's no end to the technical objections that component owner can raise against the outsider's change --- but these objections are just a smokescreen, and addressing them is futile, because the _real_ problem is that the component owner has done a thing certain way for a long time and some attempt by an outsider to shake things up triggers all the owner's ancient territorial defense instincts. The actual merit of the outsider's proposal doesn't matter: the problem is social, not technical.

In a situation like this, when the intrepid outsider does manage to exhaust the maintainer's objections, the maintainer will switch to just spewing unfalsifiable FUD, to inventing further ridiculous objections, or just simply ignoring the outsider's change entirely. There's no technical fix for a curmudgeonly maintainer who really doesn't want to entertain new ideas.

In the corporate world, to resolve this problem, you usually have to go up the management chain and explain to leadership that certain people are making important work impossible. (You should try every possible alternative means of persuasion first, of course.) In the FOSS world, you just have to fork as a last resort.

<comment><USER23> What a load of bullshit. Theo has had more staying power than the Rust community collectively to date as a maintainer of a mission critical OS. That's a lot of responsibility and I personally very much value his defense of what matters a lot to me: backwards compatibility. Not breaking stuff is the prime directive at that level. Suggesting that your rewritten, non compliant, non-drop in replacements and associated toolchain should be part of the default is hubris at a level that does not befit nor benefit the Rust community.

If you want to reach that level then you need to commit for real, and solve those issues that Theo rightfully marks. Even though this was written in 2017 it is just as applicable today: either you replacement is as good on all fronts as the thing it replaces _and_ has some additional benefits replacement of something battle tested makes no sense.</comment></comment></comment>


Show HN: My Weekend Project, a MPlayer frontend using Perl Dancer - <AUTHOR> https://github.com/Wollw/RadioDance

<comment><AUTHOR> It's really nothing special but I thought I'd post about it as Perl Dancer really ended up being a joy to work with. I've heard so much about Node.js and played around with it for awhile but for some reason I just felt like I was fighting with it the whole time (probably my lack of experience with JavaScript more than anything). In contrast, Perl Dancer turned out to be really easy for me to understand.

</comment><comment><USER2> Could you explain the project in a bit more detail? Right now I'm not 100% sure what it does.

Basically I'm confused because I can't relate to the problem. I've never needed to ssh into a server to play an internet radio station. What kind of a media setup do you have to build to reach that problem?

<comment><AUTHOR> I don't really have much of a media setup actually. I have my laptop and a phone I use to listen to music a lot but the server just happens to be in a central, permanent location and can therefore just be left plugged into some speakers. The SSHing only began recently when it occurred to me I could use server as a jukebox of sorts and for a few weeks I was just running a ssh session on my phone to play stations.

The project itself is really just a simple front end to mplayer and a sqlite database to store station info. A large part of the reason I made this was just to have something to work on (as I'm trying to do some programming every day).

<comment><USER2> Ah, OK. Thanks for the explanation. I wasn't familiar with Perl Dancer or mplayer, so things didn't really click into place until now.</comment></comment></comment>


The Perl Jam 2: The Camel Strikes Back [32c3] - <AUTHOR> https://www.youtube.com/watch?v=eH_u3C2WwQ0 <comment><USER2> Question from the Internet: "Does your exploit also work in tainted mode?"

Netanel Rubin: "No".

This simple "No" demonstrates the fundamental deception inherent in Netanel's talk.

\----

From the Perl security document[1]:

> This flag [taint mode] is _strongly_ suggested for server programs and any > program run on behalf of someone else, such as a CGI script.

\----

So:

1\. Any and all code, even in, say, Haskell, should typically take some protective measure to counter the attack potential inherent in reading in arbitrary unsafe user input.

2\. Perl provides a nice, simple, effective tool for dealing with this -- the `-T` command line option that turns on taint mode.

3\. Netanel knows about this feature and could easily have mentioned and suggested "Use Taint mode". But that would render his talk pointless...

[1] [http://perldoc.perl.org/perlsec.html](http://perldoc.perl.org/perlsec.html)</comment>


Taking the labor out of baby books - <AUTHOR> http://cloudinary.com/blog/taking_the_labor_out_of_baby_books

<comment><AUTHOR> This article describes how Baby's Firsts solved its storage and image transformation challenges with a cloud-based image management solution. [http://cloudinary.com/blog/taking_the_labor_out_of_baby_book...](http://cloudinary.com/blog/taking_the_labor_out_of_baby_books)</comment>


The Mathematical Nomad, Paul Erdős - <AUTHOR> https://medium.com/cantors-paradise/the-mathematical-nomad-paul-erd%C5%91s-3163b70a2863 <comment><USER2> There is an IMO outstandingly illustrated children's book about Erdős, "The Boy Who Loved Math" by Deborah Heiligman and LeUyen Pham [1]. Really witty, but also moving. I think it depicts really well why it makes sense to accept "strange people".

[1]: [https://www.amazon.com/Boy-Who-Loved-Math- Improbable/dp/1596...](https://www.amazon.com/Boy-Who-Loved-Math- Improbable/dp/1596433078) (I suggest Looking Inside the book as well!)

<comment><USER3> Also, for adults:

* [https://en.wikipedia.org/wiki/The_Man_Who_Loved_Only_Numbers](https://en.wikipedia.org/wiki/The_Man_Who_Loved_Only_Numbers)

Anecdote from the book: he called cats "fascists". Asked why, he replied: "Have you ever considered things from the mouse's perspective?".

</comment></comment><comment><USER4> I love the tomato juice story from The Man Who Loved Only Numbers:

> Graham was not the only one who had to put up with Erdös's kitchen antics. > "Once I spent a few days with Paul," said Janos Path, a fellow Hungarian > emigre. "When I entered the kitchen in the evening, I was met with a > horrible sight. The floor was covered by pools of blood-like red liquid. The > trail led to the refrigerator. I opened the door, and to my great surprise > saw a carton of tomato juice on its side with a gaping hole. Paul must have > felt thirsty and, after some reflection, decided to get the juice out of the > carton by stabbing it with a big knife."

[http://movies2.nytimes.com/books/first/h/hoffman- man.html](http://movies2.nytimes.com/books/first/h/hoffman-man.html)

<comment><USER5> Is that János Pach [https://en.wikipedia.org/wiki/J%C3%A1nos_Pach](https://en.wikipedia.org/wiki/J%C3%A1nos_Pach) ?

<comment><USER4> Yes you are correct, and Pach is noted as one of the most frequent collaborators with Paul Erdos. My apologies, I no longer have the option to edit my post to correct the spelling.

</comment></comment></comment><comment><USER6> Medium is now limiting the number of posts one can read before signing up with a Google or Facebook account.

Is there a reasonable workaround for this that doesn't require signing in?

EDIT: I suppose private browsing works well.

<comment><USER7> Boycott Medium and maybe eventually the pain will stop?

</comment></comment><comment><USER8> There is a wonderful book called _Proofs from THE BOOK_ , highly recommended if you like maths.

Can't say it better than Wikipedia:

> Proofs from THE BOOK is a book of mathematical proofs by Martin Aigner and > Günter M. Ziegler. The book is dedicated to the mathematician Paul Erdős, > who often referred to "The Book" in which God keeps the most elegant proof > of each mathematical theorem. During a lecture in 1985, Erdős said, "You > don't have to believe in God, but you should believe in The Book."

[https://en.wikipedia.org/wiki/Proofs_from_THE_BOOK](https://en.wikipedia.org/wiki/Proofs_from_THE_BOOK)

</comment><comment><USER9> Most people know him from:

[https://en.wikipedia.org/wiki/Erd%C5%91s_number](https://en.wikipedia.org/wiki/Erd%C5%91s_number)

</comment><comment><USER10> Given his temperament and amphetamine habit, I have long thought of him as mathematics' wierd convoluted relation of Hunter S. Thompson.

</comment><comment><USER11> "A mathematician is a machine for turning coffee into theorems"

<comment><USER12> "When I contemplated leaving mathematics to go to the Technical University and become an engineer, Erdős said: "I'll hide, and when you enter the gate of the Technical University, I will shoot you"

</comment></comment><comment><USER8> > Erdős himself has Erdős number 1. People who co-authored a paper with Erdős > have Erdős number 2. People who co-authored a paper with someone of Erdős > number 2 have Erdős number 3 and so on.

No, the Erdős number is the "distance" to Erdős, so he himself has Erdős number 0, his collaborators have 1, etc.

<comment><USER13> [https://en.wikipedia.org/wiki/Erd%C5%91s%E2%80%93Bacon_numbe...](https://en.wikipedia.org/wiki/Erd%C5%91s%E2%80%93Bacon_number)

</comment></comment><comment><USER13> In the early 90s, George Csicsery made a film about Erdős called "N is a Number" which is worth watching. You can find information on that video, along with a few lectures by Erdős and a film about his 100th birthday at zalafilms.com.

I enjoy watching Csicsery's works. For example, his film "Hard Problems" shows a respect for the IMO competition and its participants that I feel the similar BBC documentary "Beautiful Young Minds" does not.

</comment><comment><USER14> Reached my Medium limit...

<comment><USER15> Peak Medium.

<comment><USER16> Unable-to-peek Medium

</comment></comment></comment><comment><USER17> The Man Who Loved Only Numbers - and meth

</comment><comment><USER18> Typo in the last section.

> He was attending a conference when he had a heard attack

</comment><comment><USER19> Interesting</comment>


Tembo – spaced repetition memorization for everyone - <AUTHOR> https://www.tembo.app <comment><AUTHOR> I worked on this project after reading the excellent article from Michael Nielsen "Augmenting Long-Term Memory" ([http://augmentingcognition.com/ltm.html](http://augmentingcognition.com/ltm.html)). I have always struggled to remember things I learn or read in the long term. And he gives a very interesting explanation of why we don't remember things in the long term. Indeed, to remember something, our memory needs to see it again and again. That's probably why advertisers we'll show you the same ad again and again. He also introduced a technique called Spaced Repetition Memorization that allows remembering almost anything. The idea is pretty simple: you have to repeat things with larger intervals each time you succeed in remembering it. In this article, he explains how he uses Anki, a software that was built with this goal. I decided to start using it. But Anki has many features and is done for people who want to dedicate time to learn to use it. It's probably perfect for very intense users, but in my opinion, it is not a product many people would be able to use and to get used to. And that's pretty disappointing since Spaced Repetition Memorization could be useful to many people: students, people wanting to learn a new language, readers, etc... That's why I decided to buy Tembo, for my use at the beginning, and for all the people who feel the same pain. Please give me your feedbacks. This first version is very simple, but that's also the goal.</comment>


Show HN: I couldn't figure out what an Emoji meant, so I made WhatMoji.com - nk- http://whatmoji.com/ <comment><AUTHOR> I feel like [http://emojipedia.org/](http://emojipedia.org/) is a better website with more information. I've been regularly using it since Emoji have become a thing. It gives more context about the meaning of symbols, is easier to search the other way, and also shows how the image appears on various different platforms.

<comment><USER2> Emojipedia doesn't say anything about alternative meanings for the peach or eggplant emojis. I need something for more cultural meanings, too.

<comment><USER3> Neither does whatmoji, though.

Taco: [http://whatmoji.com/?i=21c542c341eb67e1cb7a4ba4a21052ff](http://whatmoji.com/?i=21c542c341eb67e1cb7a4ba4a21052ff)

Eggplant: [http://whatmoji.com/?i=376bf089b6f3e4f4eb7e9e13c4ce1cb2](http://whatmoji.com/?i=376bf089b6f3e4f4eb7e9e13c4ce1cb2)

peach: [http://whatmoji.com/?i=1633b26bb2deb796dab3660132424995](http://whatmoji.com/?i=1633b26bb2deb796dab3660132424995)

</comment></comment></comment><comment><USER4> It still doesn't really tell you what it means, just the name/annotation.

<comment> nk- True, but it does give you a hint at least. I would have never guessed that this was a shrimp for example: [http://whatmoji.com/?i=b65555fbf55380ec0970a060b2457527](http://whatmoji.com/?i=b65555fbf55380ec0970a060b2457527)

<comment><USER5> Except so many of them are culturally specific, like the horn emoji on the front page has roots in postal systems (both in Japan and Europe). This page doesn't tell you that. Even this random website does a better job of giving context: [http://emojipedia.org/postal-horn/](http://emojipedia.org/postal- horn/)

<comment><USER6> Well, Emoji are originally Japanese so you always have to keep that in mind.

<comment><USER5> The point is "what is this emoji?" Knowing emoji started in Japan still doesn't tell you what the emoji is or means!

</comment></comment></comment></comment></comment><comment><USER7> Quick tip if you are on OS X: Pressing ctrl+command+space pops up an emoji bank so you can enter them in any text field. Now you don't have to hunt for emojis to put in the website.

Love the website btw!

<comment><USER8> I tend to copy and paste from Emacs, so I can search for emoji and other Unicode characters by name with C-x 8 RET.

<comment><USER9> you can also autocomplete emojis in emacs

[https://github.com/syohex/emacs-ac-emoji](https://github.com/syohex/emacs-ac- emoji) [https://github.com/dunn/company- emoji](https://github.com/dunn/company-emoji)

and display them as well

[https://github.com/iqbalansari/emacs- emojify](https://github.com/iqbalansari/emacs-emojify)

</comment></comment></comment><comment> nk- And yes, I know I could have just looked it up in many different ways.

But any programmer knows that an elaborate solution that takes a day of programming is a lot more fun than a 1-minute fix that just solves the problem at hand!

<comment><USER10> > But any programmer knows that an elaborate solution that takes a day of > programming is a lot more fun than a 1-minute fix that just solves the > problem at hand!

Don't put that on your CV/resumé

<comment><USER11> I lol'ed!

</comment></comment></comment><comment><USER12> It would be cool if you could allow for user commenting/voting so that people could add and build up cultural context for them rather than just having the spec definition. If you got enough of a community around that it could become very useful. Kinda like Urban Dictionary for Emoji.

</comment><comment><USER13> It would be nice if it supported the ASCII text shortcuts like :poop: (see [http://www.emoji-cheat-sheet.com/](http://www.emoji-cheat-sheet.com/)).

You could also make the URL route based on the ascii shortcut as well (as previously mentioned by another user).

</comment><comment><USER14> Reading this article sent me on a whirlwind of Wikipediaing and Googling...just like Unicode itself is vastly complicated, so is the thought and design behind emojis, which have their own technical subcommittee [0], currently chaired by Apple and Google employees.

One of the best articles I came across is from Huffington Post, before the inclusion of multi-colored faces in 2015:

[http://www.huffingtonpost.com/2014/06/27/emoji- meaning_n_553...](http://www.huffingtonpost.com/2014/06/27/emoji- meaning_n_5530638.html)

I knew that trying to design universally readable iconography is an essentially impossible task in itself, but the article brings up a lot of other interesting issues, such as the worry that Apple might take it upon themselves to revamp their emojis which were designed before the "flat" days:

> _By comparison, Apple's iconic emoji are horribly out of fashion. Its icons > have shadows, depth and a three-dimensional look that's a holdover from the > Steve Jobs era of skeuomorphic design. Flat illustration free of texture or > highlights is now de rigeur. Apple has already abandoned the faux-wood > paneling and leather elsewhere in its software, and its emoji could change > at any time — potentially a rude shock for those who've come to identify > with the brand's symbols. "I kind of hope they don't [change it]," said Van > Lancker._

Even the HTML of the article itself contains interesting insights, especially about web production and the challenges of digital preservation. I thought at first that, in a decade, the opening paragraph might completely lose its meaning...but then I opened up my web inspector:

[http://imgur.com/YUVtwPe](http://imgur.com/YUVtwPe)

[0] [http://unicode.org/emoji/](http://unicode.org/emoji/)

</comment><comment><USER15> It seems to permanently (?) store all of the input text. Why?

[http://whatmoji.com/?i=da510f5876a15885fa89de8abe731d3c](http://whatmoji.com/?i=da510f5876a15885fa89de8abe731d3c)

EDIT: OK, so it stores only the emojis. The actual text is removed.

<comment> nk- Yes, it stores the list of emojis so that you can link to it. The text is never stored.

Are you concerned this might be a privacy issue? I could change it to only store searches on demand.

<comment><USER15> That was my concern at first, but I don't think it's an issue if it doesn't store the entire text.

</comment></comment></comment><comment><USER16> I ran into this problem in a side project recently and made a package to programmatically fix it: [https://github.com/NarroApp/translate- emoji](https://github.com/NarroApp/translate-emoji)

<comment><USER17> Nice work!

You could even go further using algorithms like Sense2Vec ([https://spacy.io/demos/sense2vec?%3B%29|NOUN](https://spacy.io/demos/sense2vec?%3B%29|NOUN)). They are able to figure out similarities between emojis and between emojis and words.

<comment><USER18> Instagram team did it:

[http://instagram- engineering.tumblr.com/post/117889701472/em...](http://instagram- engineering.tumblr.com/post/117889701472/emojineering-part-1-machine-learning- for-emoji)

</comment></comment></comment><comment><USER19> I believe an emoji dictionary is only helpful if emoji senders universally rely and agree upon its definitions.

I find myself confused by emojis from time to time, and for me the confusion doesn't stem from misunderstanding the emoji's expression, it stems from what the emoji means within the context of the conversation and my relationship with the sender.

Emojis can lessen ambiguity when they're used to complement a message or communicate tone. I believe they increase ambiguity, however, when they're used to replace words altogether.

</comment><comment><USER20> [http://whatmoji.com/?i=148391ab065fc0104c28dbc67913f335](http://whatmoji.com/?i=148391ab065fc0104c28dbc67913f335)

I thought this was a high five...

</comment><comment><USER21> I installed the Debian package "unicode", which will accept an emoji as an argument and print the Unicode description.

It also searches the descriptions, if given a word as an argument.

</comment><comment><USER22> I believe emoji is designed to be expressive, if we need to look it up in a emoji dictionary ,then emoji goes wrong. The fact is that everybody wants to add their favourite emoji, but more emoji added more confusion it will be make. Just imagine we lookup an emoji before we send to others, and the receiver has to look it up in a dictionary to know what the sender want to express. Why not just using the normal words we already know very well

<comment><USER23> >Why not just using the normal words we already know very well

A persons answer to that can be "I don't know. I think we should just use the normal words we already know very well" yet still have to life in the current reality of many people using emojis. Hence OP's service.

</comment></comment><comment><USER24> Good site. ..funny that programmers & cell phone carriers are re-inventing Chinese-style (pictorial language)

</comment><comment><USER25> Nice! One little thing is that it really doesn't describe skin tones well: [http://i.imgur.com/dGg1026.png](http://i.imgur.com/dGg1026.png)

</comment><comment><USER26> I wish this went deeper. Emoji are like the symbols on the alethiometer.

<comment><USER27> A+ reference. Also I would 100% download an emoji alethiometer app.

</comment></comment><comment><USER28> Ah, but what does 🈚 mean? :)

WhatMoji doesn't recognize it; Emojipedia doesn't either; and Google returns zero results. Some of the original imported Softbank characters are _arcane_. _These_ are the ones I want to look up—not because the meaning is vaguely between two possible semantic clusters, but because I literally have no idea how to even describe the character to search for it.

<comment><USER29> Maybe that's just a Chinese character? It looks like the character for "nothing" (無) enclosed in a box.

Part of the problem is that the character you pasted is made up of two code points (I don't know if I have the terminology right here):

[http://apps.timwhitlock.info/unicode/inspect?s=%F0%9F%88%9A%...](http://apps.timwhitlock.info/unicode/inspect?s=%F0%9F%88%9A%EF%B8%8F)

It's both the character plus a "variation selector".:

[https://en.wikipedia.org/wiki/Variation_Selectors_(Unicode_b...](https://en.wikipedia.org/wiki/Variation_Selectors_\(Unicode_block\))

If you search for the character by itself ( 🈚 ) you get results on google:

[https://www.google.com/search?q=🈚](https://www.google.com/search?q=🈚)

</comment></comment><comment><USER30> Just an FYI, my workplace McAfee Gateway filters this site due to "Violence"(??!). Not sure what you have to do to get off that list.

</comment><comment><USER31> This looks very similar to [http://emojipedia.org/](http://emojipedia.org/)

</comment><comment><USER32> Would be nice if you provided a bank of emojis so I wouldn't have to go to another source in order to copy and paste them in.

<comment> nk- The idea behind the site was that you would see an emoji on a website somewhere and couldn't figure out what it meant. So the main "use case" I have in mind is indeed someone copy/pasting them from somewhere else.

What would you like to do that would require a bank of emojis on the site itself?

<comment><USER33> A bank would be useful for general learning. i.e. when I went to the site just now, I didn't have a specific emoji that I needed to identify, but if there had been a 'bank of emojis' I probably would have wasted a few minutes of my day reading about what they mean...because I frequently have no idea what emojis mean...

And by wasted I mean I wouldn't have been working. Not a jab at your project :)

<comment> nk- I see - I just added links to lists of most emojis to the header: [http://whatmoji.com/](http://whatmoji.com/)

I hope this helps you waste a few minutes! :)

<comment><USER34> I'm not really up on how text rendering works; what's it mean if I can see some of those, but not others? e.g. I have Rocket but not Helicopter; not Steam Locomotive but Railway Car and High-Speed Train; Taxi and Automobile but not their Oncoming variants; Toilet, Water Closet, and Bath but not Shower.

Is there something I need to install, or something? I'm on Win7 with up-to- date Chrome and Vivaldi, and they're both the same way.

<comment><USER35> You would need to have an up-to-date font that contains glyphs for these specific emoji - your computer gets an integer key (a code-point) and needs to have a scalable image in its lookup table to substitute for it. I have _no_ idea how exactly to get a new one for Windows, sorry :-\

</comment></comment></comment></comment></comment></comment><comment><USER36> I expected this to be the solution for my broken Linux font experience. In Linux there is no support for Emoji and every time someone uses them in a website to display so semi-important idea I get frustrated because the only thing I see is a blank character or a rectangular border. I thought this service was something like "discover what this non-supported character actually is" more than "discover what is the meaning of this character".

I know about the existence of alternatives [1][2] but I wish there was native support for this trend in Linux. And yes, I know emoji is actually an old thing from Japan companies, but is still a young trend in terms of western usage.

I wonder how difficult would be to copy all those images from the Apple font and create a custom TTF so I can get the fancy colored symbols in my computer. Of course, distribution of that file would be against Apple's copyright (or maybe not, I don't know) but I don't care.

EDIT1: Actually this is what I was expecting to be; I forgot I had disabled JavaScript in my browser. Still would like to have an official font with Emoji support for Linux.

EDIT2: Thanks to gtk40 I found [3] now I can create my own font with those images.

[1] [https://github.com/eosrei/emojione-color- font](https://github.com/eosrei/emojione-color-font)

[2] [https://github.com/eosrei/twemoji-color- font](https://github.com/eosrei/twemoji-color-font)

[3] [http://emojipedia.org/apple/](http://emojipedia.org/apple/)

<comment><USER37> > _I thought this service was something like "discover what this non-supported > character actually is" more than "discover what is the meaning of this > character"._

I don't see how when the title starts "I couldn't figure out what an Emoji meant"

<comment><USER36> You are right, I spoke too fast.

If I could downvote my own comments I would do so.

</comment></comment></comment><comment><USER38> whatmoji.com/[the emoji here]

would have have been cool.

<comment> nk- Cool idea, this should work now: [http://whatmoji.com/](http://whatmoji.com/)

edit: Looks like HN strips emoji so I can't link to it, but you can try it out by visiting [http://whatmoji.com/TEXT_HERE](http://whatmoji.com/TEXT_HERE)!

<comment><USER21> You just need to encode the URL for HN to accept it

[http://whatmoji.com/%F0%9F%8C%8D](http://whatmoji.com/%F0%9F%8C%8D)

</comment></comment></comment><comment><USER39> And it could work also for emoticons. For example ":P" is hard to understand. Good idea

<comment><USER40> Yeah, it's totally hard to understand :P

</comment></comment><comment><USER41> That this website exists probably says enough. I've never heard anyone ask questions about smileys like MSN or Skype had/have, just those "emoji"s are often ambiguous and usually too small to see properly from a distance.

</comment><comment><USER42> I like it. It's simple & elegant and does exactly what it promises to do. :)

</comment><comment><USER43> It would be more interesting if the emoji descriptions were user-generated, because individual subgroups often use the same emojis to mean different things (or an emoji just takes on a popular meaning of its own).

</comment><comment><USER44> There was this project called emoji.js I think that you embed in your site and transforms all emoji it sees into Apple Emoji, the standard, and works for everyone even without emoji installed. give it a shot.

</comment><comment><USER45> Cool but you definitely need to include "alternative" meanings. Consider doing something like Urban Dictionary so people can find out what the eggplant _really_ means.

</comment><comment><USER46> The keyboard on my previous phone, a z3, offered emoticons as part of suggestions while typing.

That was a nifty feature for a stock keyboard.

<comment><USER47> Google Keyboard just started doing that to me, and I can't figure out how to turn it off...

</comment></comment><comment><USER48> Here the ones I thought were for sex positions are astrological symbols.

</comment><comment><USER49> We are literally reinventing Chinese/Japanese/et all :)

<comment><USER50> Those are not pictograph languages.

<comment><USER51> But at some point Chinese probably was.

</comment></comment></comment><comment><USER52> It's amusing that people try to replace phonetic spelling with pictograms, and rediscover that phonetic spelling is better.

For one thing, you can look up spelling in a dictionary. How do you look up a pictogram?

</comment><comment><USER53> Edit: seems like HN doesn't let you emoji

</comment><comment><USER54> Urban Dictionary for Emoji?

</comment><comment><USER55> I want something like Google Translate, that will take any arbitrary text and turn it into Emoji. Make it so.</comment>


HP: To Save The webOS Dev Community, Open-Source Enyo Now - <AUTHOR> http://funkatron.com/posts/hp:-to-save-the-webos-development-community,-you-must-open-source-enyo-now.html <comment><USER2> "...one day your washing machine, stove, or even car infotainment system will run on webOS"[1]

If this is truly the direction they're heading then webOS is dead to me anyhow. Any further courting of developers is just salt in the wound.

edit: Don't get me wrong, I'm all for open sourcing Enyo and webOS, I just get the complete opposite vibe from HP's actions this week. They've basically said "screw you" both to die hard consumers and developers of the webOS platform.

[1] [http://techcrunch.com/2011/08/16/hp-looking-to-appliances- au...](http://techcrunch.com/2011/08/16/hp-looking-to-appliances-autos-to- grow-and-save-webos/)

<comment><USER3> I thought most of webOS was built on somewhat high-level web technologies like node.js? Why would they use something like that in embedded systems?

<comment><USER4> > I thought most of webOS was built on somewhat high-level web technologies > like node.js?

Yes. Enyo is webOS's second-generation SDK and development framework (replacing the original Mojo), and it's a javascript framework.

FWIW, during the original presentation of Enyo (late 2010) it was described as "for resource-constrained devices" where "resource-constrained devices" are the original Pres: 500MHz CPU core and 256MB RAM.

</comment></comment></comment><comment><USER5> Open souce isn't the only possibility.

I think HP should skunkworks it. Round up all the engineers who developed the platform, form an independent LLC, retain an equity stake, and let them proceed with full autonomy.

Tablet and smartphone hardware is going to get so dirt cheap in just a few years, that they'll probably be able to outsource all the hardware and run the operation like a scrappy startup.

<comment><USER6> Who supplies the funding for it?

HP has already made it clear that they won't put in another dime.

The problem with hardware is that the enterprise won't buy without a big name backing it as it makes it alot less likely that the brand will be around.

I'd like to think that your idea will work.

What is your "secret sauce" that will make this work that didn't with Palm, HP, the crunch pad, newton, Go tablet, etc?

<comment><USER5> Funding isn't a problem, there has never been a better time to raise money than right now. If WebOS could become the premier development platform (develop on WebOS, deploy to the web and others platforms via Phonegap) you could gain a following.

<comment><USER6> > Funding isn't a problem,

That's a pretty big assumption but lets say that's true. I don't doubt that they coudl raise some money, I'm just not sure that a WebOS company could raise enough to fund a complete hardware and software product through to finish.

The biggest reason I think this wouldn't happen is that I can't see HP wanting to assign any patents to this company as patents have significant value, as we've seen of late.

Who would support a company that doesn't own any sort of patent portfolio to defend itself?

</comment></comment></comment></comment><comment><USER7> In case people were wondering what Enyo is, it's HP/Palm's Framework for building WebOS apps (or generally mobile apps using HTML).

<comment><USER8> ... and it is just JS/HTML/CSS so it will run on any webkit browser and you can use it to write nice-looking mobile apps for iPhone, Android, etc...

</comment></comment><comment><USER9> Guys... The idiot who killed WebOS came from SAP. If you think Microsoft is evil, they are 10 times worse.

> Release Enyo under a liberal open source license (Apache, MIT, etc)

Why not GPL? That would ensure no competing platform could grab it and run.

<comment><USER10> >Why not GPL? That would ensure no competing platform could grab it and run.

Not that I don't like GPL, but why would HP care at this point? They just took their only viable product and said "Whatever". They act like they have an interest in licensing webOS. Ha! How long have they been shopping licensing? Even if it is true, they've just coated it in a nice splash of red paint.

<comment><USER9> > but why would HP care at this point?

The GPL would ensure their competitors can't keep their improvements to themselves. But I agree - the odds of a former SAP kahuna understanding how to leverage open-source are very close to zero.

</comment></comment></comment><comment><USER11> Also, if you're feeling generous, open-source webOS too please? :)

</comment><comment><USER12> I have to say that I was really impressed by the Netflix and Mail demos in this talk.

<http://www.youtube.com/watch?v=nsZX2dJW5Ss>

Assuming the speed was there, and it will be as hardware improves, WebOS could take over as a cross-platform rapid development environment.

If this was open-sourced and made to look platform native on the iPad, I would use it for sure.

</comment><comment><USER10> Please, please do this HP. WebOS is really impressive in terms of technologies in my opinion. You can build entire Enyo apps in your browser, add a few hooks for WebOS services, and you're done. It's crazy easy, and Enyo makes it equally easy to have one app that works well on mobile, tablet and desktop devices. It's a shame to see HP pull the rug out from under it, but I'm really hoping they open source Enyo or better webOS.

</comment><comment><USER13> Meh, I would just be fine with them allowing it on other platforms. Don't really care how they license it to make that happen.</comment>


Learning Diverse Skills via Maximum Entropy Deep Reinforcement Learning - <AUTHOR> http://bair.berkeley.edu/blog/2017/10/06/soft-q-learning <comment><USER2> DeepMind's Combining Policy Gradient and Q-Learning provides good historical background:

[https://arxiv.org/pdf/1611.01626.pdf](https://arxiv.org/pdf/1611.01626.pdf)

And archived videos from Berkeley's Deep RL Bootcamp is a crash course to get one up to speed:

[https://sites.google.com/view/deep-rl- bootcamp/lectures](https://sites.google.com/view/deep-rl-bootcamp/lectures)

But consider the multi-model Q-distribution in Fig. 3. That graph, and the variance of its peaks, is the key to understanding why training Deep RL algorithms becomes so intractable in stochastic, real-world settings.

Max Entropy Estimation and approximate Bellmen Equations will do very well in Atari games for example. Or creating unbeatable MOBA competitors. But for robotics a breakthrough that allows for quick generalization from sparse input data is what is needed.

One-Shot Visual Imitation Learning via Meta-Learning

[https://sites.google.com/view/one-shot- imitation](https://sites.google.com/view/one-shot-imitation)

<comment><USER3> But what approach would it take to beat a human at Ms. Pac Man?

When people hear ML has felled the worlds classic strategy games like Chess and Go, leaving world champions helpless, the idea of humans beating computers any game seems quaint.

But forget world champions, the best systems cannot beat a _mediocre_ player at Ms. Pac Man. Its not because no one has tried. There have been some competitions and research papers.

Of course, the challenge seems pretty damn unimportant compared to self driving cars. But still, if it were trivial to do, I would have guessed someone would have done it.

What other games exists, where AI can't beat humans with less than 60 minutes of lifetime experience?

<comment><USER4> Is that true about Ms. Pac Man? Microsoft seems to have posted near perfect score:- [http://www.businessinsider.com/microsoft-ai-ms-pac-man- highe...](http://www.businessinsider.com/microsoft-ai-ms-pac-man-highest-all- time-score-2017-6)

<comment><USER3> Wow, that's really new, within the last few months, thanks for the link. The challenge sounds smaller than it was. It had been worked on for years prior, as I mentioned with poor results.

Now on to Starcraft 2, where some major research labs are competing including DeepMind.

It looks like the best bots this year struggled to beat average players. [https://www.wired.com/story/facebook-quietly-enters- starcraf...](https://www.wired.com/story/facebook-quietly-enters-starcraft- war-for-ai-bots-and-loses)

<comment><USER5> I am somewhat curious which game you will be latching on to when SC2 is also thoroughly beaten in a year (or whenever).

<comment><USER3> "latching on to"? It's not about trying to disrespect the research, or downplay the capabilities of AI after the last decade's amazing progress.

It's about noting all the little milestones and unsolved problems along the way, not just the headlines. The idea is to improve perspective on the whole of where the state of the art is.

As for the next game, I don't imagine it will be too many years before we reach the 99% percentile of AI playing all games at least as well as the best humans. Why not throw out a prediction? Less than 5 years?

</comment></comment></comment></comment></comment></comment><comment><USER6> I fail to see how this is different than keeping non-zero weights on suboptimal actions for exploration?

<comment><USER7> In standard policy gradient, the weights for (apparently) suboptimal actions are free to trend downwards to arbitrarily small values. If you're using a Gibbs distribution[0] the probability of executing alternative actions is nonzero but potentially very small.

With entropy methods, you add an artificial bonus towards policies that execute more varied actions[1]. So while there's a tendency towards improving the policy by executing apparently favorable actions, it is counter-balanced by a term incentivizing different choices.

Why not just optimize using standard gradient descent? Typically, the variance is high and the algorithms optimize very greedily, so you end up in situations where a few bad trajectories cause certain actions to be avoided. Or the agent might need to refine its policy for the states that come afterwards, but because that hasn't happened yet, the actions that lead to those states are ignored[2].

So exploration needs to be incentivized, and while this can be done with ε-greedy or optimistic initialization, those techniques are kinda ad hoc, while entropy methods have a bit more mathematical/philosophical heft behind them. Plus they seem to work, which is always nice.

\-----

0\. That is, if you'll excuse the notation: p(a) = e^α/Σ e^β where 'p(a)' is the probability of taking action 'a', with weight 'α' and you're summing over all possible actions (with weights denoted 'β') in the denominator.

1\. For a discrete space (absent other information), the maximum entropy distribution is the uniform distribution, i.e., all actions are equally likely.

2\. For example, a human-like robot might try stepping forward, but hasn't learned how to balance after it lifts its foot, so it stumbles and falls. Crawling is less efficient, but provides constant progress and is easier to learn. So the robot 'gets stuck' crawling since it had a bad experience walking and doesn't attempt it again.

</comment></comment><comment><USER8> Could someone please explain the explanation of equ. 18 in the appendix to the paper? It seems key, but isn't clear to me.

> if one greedily maximize the sum of entropy and value with one-step look- > ahead, then one obtains π˜ from π

<comment><USER9> Hi Alex! I don't have the answer, but I just sent this over to Seth. Looking forward to seeing you all again in December!

<comment><USER8> Hey, Skander! Hope SF's treating you well.</comment></comment></comment>


How to Cover the One Percent - <AUTHOR> http://www.nybooks.com/articles/2016/01/14/how-to-cover-the-one-percent/ <comment><USER2> I'm ambivalent about this article. While I agree that the hidden power of mega-philanthropists is unsettling, the hidden players of our "public" government are orders of magnitude more powerful.

Frankly, differentiating between the "public" and "private" when discussing the ills of the %1 is pointless [1]. People at this level of society move in and out of the top levels of corporations and government all the time. They socialize together, intermarry, and share family bonds. Only together have big business and big government created this massive feedback loop that promotes inequality and loss of freedom for the common man.

It pains me to see my "right" and "left" leaning friends blame one side of the coin while completing ignoring the damage wrought by the flip side.

1\. To be clear, the top 0.01% is what the article is really referring to, since the top %1 includes people like successful everyday doctors and lawyers. Perhaps a small cadre of powerful bureaucrats at this income level could be reasonably included in the top tier group the article refers to, but for the most part, these are people who are wealthy and influential in their local communities, but decidedly not actors at the level the article describes.

<comment><USER3> >It pains me to see my "right" and "left" leaning friends blame one side of the coin while completing ignoring the damage wrought by the flip side.

George Washington, Thomas Jefferson, John Adams, and many others explicitly warned about this.

>The alternate domination of one faction over another, sharpened by the spirit of revenge, natural to party dissension, which in different ages and countries has perpetrated the most horrid enormities, is itself a frightful despotism. But this leads at length to a more formal and permanent despotism. The disorders and miseries which result gradually incline the minds of men to seek security and repose in the absolute power of an individual; and sooner or later the chief of some prevailing faction, more able or more fortunate than his competitors, turns this disposition to the purposes of his own elevation, on the ruins of public liberty. Without looking forward to an extremity of this kind (which nevertheless ought not to be entirely out of sight), the common and continual mischiefs of the spirit of party are sufficient to make it the interest and duty of a wise people to discourage and restrain it.

-George Washington, Farewell Address

It's the age old strategy of divide et impera. It couldn't work if the masses weren't arguing futilely among themselves:

>Now, what does all of this mean in this great period of history? It means that we've got to stay together. We've got to stay together and maintain unity. You know, whenever Pharaoh wanted to prolong the period of slavery in Egypt, he had a favorite, favorite formula for doing it. What was that? He kept the slaves fighting among themselves. But whenever the slaves get together, something happens in Pharaoh's court, and he cannot hold the slaves in slavery. When the slaves get together, that's the beginning of getting out of slavery. Now let us maintain unity.

-MLK, I've Been to the Mountaintop. Last speech given the night before assassination.

<comment><USER4> I consider myself a skeptic. To me JFK was killed by a lone gunman, we went to the moon, and 911 wasn't a inside job. Then I read I've Been to the Mountaintop and wonder if there wasn't some involvement, at some level, from the US government (or those who control it) in MLK's death. It is already proven they attempted to discredit him, did they simply take that one step further?

Then I look at stuff like black lives matter. More white people were killed by police last year than black people. What did all the people murdered by police have in common? They were male, and they were poor. Are we sure this is a white/black problem and not a rich/poor one? Framing it in the black/white context seems to keep the for profit prisons humming along happily.

<comment><USER5> There's a kernel of truth here, but you have to be careful about comparing _numbers_ of white/black people killed/jailed. You need to look at the per- capita (percentage) killed or jailed. There are way more "white" people so of course there are more of them killed/jailed/whatever! But when you look at the percentage then you see the injustice. "The country is about 63 percent white and 12 percent black... death rate due to legal intervention was more than three times higher for blacks than for whites in the period from 1988 to 1997."

Source: [http://www.politifact.com/punditfact/statements/2014/aug/21/...](http://www.politifact.com/punditfact/statements/2014/aug/21/michael- medved/talk-show-host-police-kill-more-whites-blacks/)

<comment><USER4> I understand that but black people have statically lower incomes and that might be a reason that they are over represented. What you never see is police killing women or the wealthy of any color.

</comment></comment></comment></comment></comment><comment><USER6> "Ten years ago, for example, Google had a one-person lobbying shop in Washington; today, it has more than one hundred lobbyists working out of an office roughly the size of the White House. " ... to the tune of $16.8 Million dollars.

[https://www.opensecrets.org/lobby/clientsum.php?id=D00002200...](https://www.opensecrets.org/lobby/clientsum.php?id=D000022008)

I really like this article. It's a well-researched call to action for journalists to do real investigative work.

</comment><comment><USER7> Top 1% is (based on [http://money.cnn.com/calculator/pf/income- rank/](http://money.cnn.com/calculator/pf/income-rank/)) >$400k / year income. The billionaires this article talks about are in much, much higher brackets - maybe something like top %0.01

* P.S. based on another source [[https://www.washingtonpost.com/news/get-there/wp/2015/01/26/...](https://www.washingtonpost.com/news/get-there/wp/2015/01/26/what-the-top-1-percent-makes-in-every-state/)], it's state-dependent and the highest I see is $678k in CT

<comment><USER6> 1% in journalism = the elite wealthy controlling banks, businesses and bureaucrats.

</comment></comment><comment><USER8> Summary: The wealthy are using non-profits, charities, and foundations to push political agendas at the expense of taxpayers who are footing the bill because these institutions are also tax write-offs.

The article focuses on these organizations pushing advocacy, but I would also add that many "non-profit" organizations are actually incredibly profit- focused. An early introduction to this for me was growing up next to the Christian Broadcasting Network, a charitable organization that was raking in millions of dollars selling alternative medicines, opening five-star restaurants, hosting luxurious retreats, while also raking in donations from the poor people who would tune into the show seeking god's grace [1].

I see the same thing with Mega Churches, which are springing up like fast food franchises all over the DC-Metro region. These enormous buildings are raking in profits every Sunday while giving nothing back in taxes to pay for the roads, utilities, and emergency services the community provides for them. Taxpayers are losing $71 billion a year in revenues because of these for- profit franchises [2].

It isn't just religious institutions either. I've stopped giving blood to the American Red Cross in favor of donating it directly to my local hospital so they don't have to pay the overhead that comes with Red Cross blood [3]. The DC-Metro area is packed with "non-profits" that are really for-profit companies that pay out their profits directly to CEOs and board members while sending lobbyists to the National Mall to buy expensive lunches for our Representatives.

The article notes that the alternative is the European system of not providing tax-breaks for these organizations--which results in much fewer organizations. It's hard for me to accept that when I think of all the organizations I _perceive_ as doing good in America, but maybe they aren't really doing good and are simply enjoying the good sentiments that come with the words "charity" and "non-profit?"

[1] [http://www.starnewsonline.com/article/20070327/NEWS/70327037...](http://www.starnewsonline.com/article/20070327/NEWS/703270372?Title=Lawsuit- says-evangelist-Pat-Robertson-abuses-tax-free-status-to-push-product)

[2] [http://bigthink.com/21st-century-spirituality/how-to- make-71...](http://bigthink.com/21st-century-spirituality/how-to- make-71-billion-a-year-tax-the-churches)

[3] [http://www.nytimes.com/1991/07/07/business/all-about- blood-b...](http://www.nytimes.com/1991/07/07/business/all-about-blood-banks- a-multibillion-dollar-business-in-a-nonprofit-world.html?pagewanted=all)

<comment><USER9> > The article notes that the alternative is the European system of not > providing tax-breaks for these organizations--which results in much fewer > organizations. It's hard for me to accept that when I think of all the > organizations I perceive as doing good in America, but maybe they aren't > really doing good and are simply enjoying the good sentiments that come with > the words "charity" and "non-profit?"

You've got to ask how much of those sectors' income is going to organizations that are doing good, and how much is not. And remember that taxes are used for the public good, so the bar to be tax exempt should not be merely "is this organization doing good?" but rather "is this organization doing more good per dollar than is done with general taxation?"

<comment><USER10> Is the government "doing good?" There is some good, but there is a case that non-profits such as the Red Cross, churches, food banks, are more accountable and do more good.

To me non-profits do far more good per dollar than the government does through general taxation. The biggest reason why is that non-profits have a much more defined scope and mission.

<comment><USER11> I find it _very_ difficult to believe that the Red Cross does more good than Medicare and Medicaid, or that food banks do more good than Food Stamps.

<comment><USER12> You're comparing an organization with a budget of about $3 billion to one with a budget of over $1 TRILLION. It's a nonsensical comparison.</comment></comment></comment></comment></comment>


Show HN: Automated Follows Ups on Slack - <AUTHOR> https://www.getitdonezo.com <comment><AUTHOR> Co-creator here. We built Donezo because we know how annoying it is to assign your tasks to your team, and then have to follow up repeatedly. Donezo automates this by taking the idea behind Slack reminders and following up with each person individually before the due date. It lets you know who has completed each task and who you're still waiting on.

Right now it's in free beta, so we're really just trying to understand how can simplify existing workflows. Happy to answer any questions about how it works!</comment>