Structural and semantic deficiencies in the systemd architecture - vezzy-fnord http://blog.darknedgy.net/technology/2015/10/11/0/ <comment><USER2> Since it looks like this article is trying to treat a controversial topic substantively, let's see if we can do the same. If you comment, please stick to the substance and avoid dismissive comments and rudeness. We should all do so anyway, but when the topic is polarized it can take more conscious effort.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

</comment><comment><USER3> This is a very detailed article, and I appreciate it and will need to study it carefully.

But I think that the major argument it is making is "systemd is a mediocre service manager; similar tools existed before systemd, and were better." That is a very different argument from "we would be better off with sysvinit," which seems to be the primary argument I see in practice: if you're arguing things like "do one thing and do it well," "shell scripts are great and debuggable," "init should reap children and nothing else," "there's too much code in pid1," "I don't want to learn something new," etc., you're making that form of argument.

If there is general consensus from studied opponents of systemd that something systemd- _like_ is an obviously good idea, great. Let's move on, acknowledge the network effects of systemd, and build something that can feasibly be the shinier, newer, better, securer systemd. Which might even be based on an existing system like s6. But then the path forward has to be either improving systemd or replacing it, not continuing with sysvinit.

I am in practice a pro-systemd person, but that's only because my realistic alternatives, for my use case, are sysvinit and Upstart. I have no idea how I'd start to build a system that used something like s6. I would happily admit there is extensive room for improvement in systemd, but we UNIX folks have been working with worse-is-better and rough-consensus-and-running-code for decades.

If the general consensus is that a systemd-style service manager is great, but it should not be pid 1, that's an argument worth making, but I don't see much focus in this article on that.

If the general consensus is that everything can be done with sysvinit, this article doesn't make that case at all. (But I suspect that's not the general consensus.)

<comment><USER4> Talking about "general concensus" is hard because there are a lot of different viewpoints, and depends on how much of the debate one is exposed to.

I have seen people compare systemd unfavourably to daemontools, s6, runit, Solaris' SMF, and Apple's launchd, to mention a few. I've never seen anyone so reactionary that they suggest that SysV is in any way or shape the _better_ alternative. A lot of people are unhappy with SysV, but if they don't like systemd, they would probably consider sticking with SysV to be the lesser of two evils.

> Let's move on, acknowledge the network effects of systemd, and build > something that can feasibly be the shinier, newer, better, securer systemd.

Unfortunately, I think that for those who don't like systemd, their criticisms (its design is a monolithic, intrusive, all-consuming, all-or-nothing ball of yarn contrary to Unix philosophy about small, composable programs cooperating over standard protocols) cannot be solved by improving systemd itself, at least not without a huge undertaking to redesign and break it apart, somehow.

<comment><USER5> > I've never seen anyone so reactionary that they suggest that SysV is in any > way or shape the better alternative.

Unfortunately it's... a position that is not as rare as it ought to be. Some of the frustration and heat undoubtedly comes from the fact that while reasonable people can disagree about the best way to implement many of the goals of systemd, it's hard to take anyone seriously when they propose that SysV init is "more Unix" or technically better.

> ...cannot be solved by improving systemd itself

I suspect the most fundamental impediment is that upstream simply aren't interested in radical changes to how it works - it would be like trying to convince Linus that Linux needs to be a microkernel, or Theo that he needs to start using Rust for OpenBSD.

Any replacement for systemd is probably going to have to do what systemd did (and what daemontools and runit didn't achieve) and create a broad buy-in, with the additional overhead of working with at least some of the systemd dbus interfaces for compatibility.

<comment><USER6> It's important to realize that "a replacement for systemd" is neither achievable nor desirable.

systemd has grown fast and conquered large market shares because it was backed up by a company, which put a lot of money and manpower into it - to write it, to integrate it, to advertise it. The only thing that has not been correctly funded in systemd is research and design.

systemd is only Open Source if you read the license; all its other aspects are proprietary - it's software made by a company and aiming to capture a market. It is impossible to compete with systemd on the same grounds, because no real Open Source developers will have as many resources as Red Hat.

And even if it was possible, the result of such an attempt would simply be another integrated behemoth, powered by money and marketing instead of good technical decisions. (Or even worse, powered by ideology - can you imagine a systemd-like controlled by the FSF?) In the end, the situation for the users would be even worse than it is today. You don't fight Goliath with Goliath; you don't fight Microsoft's hegemony by buying Apple products.

About interface compatibility: the author of the DnE article (vezzy-fnord) has written uselessd, and finally abandoned the project because the systemd interfaces are so tightly integrated with the systemd design in the first place that it's impossible to be compatible without simply being a systemd clone, which he did not want uselessd to be. No, interface compatibility is not an option, because it would simply acknowledge the validity and superiority of the systemd architecture, and nobody wants a copy of systemd.

I believe that the way to provide an alternative to systemd is to provide all the functionality that the systemd users like, but in a technically better, less integrated, more unixish way.

With s6, s6-linux-init and s6-rc, I now have respectively a process supervision system, a simple init process and a service manager, which should be sufficient for a majority of applications. The next important thing that sysadmins like in systemd seems to be cgroup management, so I'd like to study the thing soon and assess what needs to be done next; but for now, I believe that the s6 family of programs is now viable as a serious alternative to systemd, and I would love to give it a broader audience.

<comment><USER7> > systemd has grown fast and conquered large market shares because it was > backed up by a company, which put a lot of money and manpower into it - <AUTHOR> > write it, to integrate it, to advertise it. The only thing that has not been > correctly funded in systemd is research and design.

> systemd is only Open Source if you read the license; all its other aspects > are proprietary - it's software made by a company and aiming to capture a > market. It is impossible to compete with systemd on the same grounds, > because no real Open Source developers will have as many resources as Red > Hat.

This. The reason systemd is hated so much by "old unix guys" is because it shows that Red Hat basically owns linux the way Microsoft owns Windows and Apple owns OS X.

Many people protested against systemd but it was still pushed by force. It was made specificaly incompatible to make experience on BSDs worse and it sucessfuly removed Gnome from there.

So yeah. Some "haters" talk about unix this and complex that, but I believe it's mostly about scummy company showing off it owns linux now.

<comment><USER8> I would say it's more like RedHat has actually stepped up and made a "Linux- derived OS" the way that Android is a "Linux-derived OS" or OSX is a "BSD- derived OS." They've thrown the "basic Unix principles"—the ones that make Linux distributions effectively interchangable commodities, but with software having to target a _very low_ lowest-common-denominator of functionality—to the side, and instead adopted a complete "base system API", in the same sense that CoreFoundation+Cocoa is for OSX, and Win32 is for Windows.

As with those other APIs, you can now "target systemd" in the same way you could "target Win32" or "target Android"—and, because of this, we'll probably see a virtualization library (like winelib is for Win32; like ARC is for Android) spring up, giving other OSes the ability to provide the "systemd API" without needing to run a Linux kernel.

From RedHat's perspective, they were just trying to copy OSX and Android: making a "real OS" atop Unix-y foundations. The interesting thing is that a large number of other providers in the Linux space has started to agree with their design decisions—effectively unifying on the "systemd platform" rather than on the "Posix platform" they were focusing on previously.

I think this has been coming for a long time; the large OS providers already think of Desktop Environments as the GUI equivalent to a "platform" rather than as application software for the OS; thus why Ubuntu, for example, has a separate distribution for each GUI software suite.

In a discussion of systemd history held here previously, it was pointed out that back in 2007-or-so, there was a large amount of interest expressed by various distro makers in adopting OSX's launchd as the basis for a new platform standard, if only it had been FOSS at the time. Early systemd was a launchd clone; upstart also started as a launchd clone before mutating.

I have a feeling that although RedHat was the one to step up and _create_ a platform, _all_ of the major distro providers were actually waiting to hop on the first available FOSS "OS platform", no matter who it came from. If, for example, Apple had created "Apple Linux": a version of Linux that runs the OSX GUI and OSX apps—then I can bet you right now that every other distro would have standardized on whatever "platform" gunk was in-between that GUI and the Linux kernel; CoreFoundation et al would have become to Linux distros as WebKit is to mobile web browsers.

As it stands, it's only an accident of history that this didn't happen with the _Android_ platform. (If the Android Runtime ran native-speed binaries from the start, and the GUI was developed five-or-so years later such that it was developed from the start with Win8-like unified desktop/mobile support in mind rather than getting type-cast as "only for mobile devices", I could bet that Linux would "be" Android now.)

\---

With all that said, while all the _big, commercially-backed_ Linux providers have been waiting on tenterhooks for a "platform" to latch onto, all the little distros are looking to remain Unix-y, and that's a wonderful thing.

Think about it: when you've got the POSIX platform "for desktops and servers" and the Android platform "for mobile", and nary the twain shall meet, it makes little sense to care about cross-compatibility; why would you want to run Android apps natively on your desktop? But if you add in a third platform that's _also_ for desktop—the systemd platform—suddenly kernel developers and distro makers and whatever you call the FreeDesktop people start thinking about how to create a Linux-based OS that natively supports _multiple_ platforms, so that it can run code targeting systemd, _and_ code targeting Plain-old-POSIX, without making a mess. And once you have that infrastructure in place... why not let it run code targeting Android too? Why not first-class Wine support? Etc.

One thing Windows has had forever (though lately in a degraded state) is multi-runtime (or, given the above discourse, multi-"platform") support. Windows can be Win32 to one program, OS/2 to another, and POSIX to a third. OSX had the capability for multi-runtime support back when it was called Rhapsody (the "Red Box, Blue Box, Yellow Box" architecture) and also co- supported Cocoa and Carbon apps for about a decade. And these are just single companies trying to meet small needs. Linux, the OS with a kernel that supports every device a nerd ever cared to packet-probe, could easily become "the multi-runtime OS", if we care enough to make it that way.

If we don't, though, at least systemd is decent. :P

<comment> vezzy-fnord _In a discussion of systemd history held here previously, it was pointed out that back in 2007-or-so, there was a large amount of interest expressed by various distro makers in adopting OSX 's launchd as the basis for a new platform standard, if only it had been FOSS at the time._

I don't think there ever was a _large_ amount of interest. In fact, the only party that studied launchd at the time (circa 2006) was Canonical. It was indeed licensing issues that stalled further research, but there was also another contender at the time being considered called initng, which Canonical ended up rejecting and went on to write Upstart instead, led by Scott James Remnant. See SJR's proposal and introduction to Upstart. [1]

 _Early systemd was a launchd clone; upstart also started as a launchd clone before mutating._

Upstart was never a launchd clone to the best of my knowledge. launchd was likely a spark that influenced Canonical to take action, but the design is pretty different.

Your musings about the "multi-runtime" convergence that systemd will allegedly enable do not appear to pan out. systemd is nothing like the OS X Core frameworks or like the Windows Runtime, it's much lower level than that. It's more of a middleware than a runtime platform (think Hurd, not Core Foundation).

There is also absolutely nothing implying that GNU/Linux will ever target Android code. The divergences are plentiful, with my article about an Android init porting attempt listing only a few. [2] Nor is there any expressed interest from any Android vendor to have any serious GNU/Linux convergence to begin with.

Not sure what "first-class Wine support" is supposed to mean. Wine is pretty self-contained.

 _If we don 't, though, at least systemd is decent._

You're too big of a dreamer, I'm afraid.

[1] [https://wiki.ubuntu.com/ReplacementInit](https://wiki.ubuntu.com/ReplacementInit)

[2] [http://blog.darknedgy.net/technology/2015/08/05/0-androidini...](http://blog.darknedgy.net/technology/2015/08/05/0-androidinit/)

<comment><USER8> > systemd is nothing like the OS X Core frameworks or like the Windows > Runtime, it's much lower level than that. It's more of a middleware than a > runtime platform

You're not considering the full scope of the thing labelled "systemd." If you only use parts of it, it's middleware, yes. If you "work with the grain" of systemd, though, then you're packaging services in nspawn containers and so forth, which _does_ constitute a separate "platform target", in the same way that e.g. CoreOS is a (mostly-ignored attempt at a) "platform target."

Basically, I'm talking here about a Linux equivalent to Solaris's "Branded Zones": within a container boundary, an app can be made by the system to think it's running on "systemd Linux", or on "POSIX Linux", or on "Android Linux", or on BSD or Win32 or Cocoa or whatever else. Runtime-virtualization is done at the system container-management level (with more or less help from the kernel), rather than expecting applications to "port" themselves by applying their own proprietary virtualization wrappers ala Cider for OSX.

> Not sure what "first-class Wine support" is supposed to mean. Wine is pretty > self-contained.

I mean, basically, support to the level of DOS applications run in Win32 VMM containers, rather than to the level of X11 applications run on OSX: management of Wine sandboxes as OS-level "runtime containers", such that you could run and _maintain_ Wine apps alongside other apps, in production, using the OS's maintenance tooling. A Linux "ReactOS Runtime" equivalent to Windows's "POSIX Runtime."

> Nor is there any expressed interest from any Android vendor to have any > serious GNU/Linux convergence to begin with.

I don't mean to suggest convergence. This is something very different: making "running an Android binary" a primitive action of _the system_ (not the Linux kernel), where "using an Android virtualization layer" is then an _implementation detail_ of how the system accomplishes this. Think of Linux's binfmt_misc and its ability to execute e.g. JVM code by booting a JVM, but with an upcall to the init daemon to decide how to implement the policy of running a particular binary format. If that init daemon is effectively a container-manager that understands how to instantiate and manage different "branded containers" for each runtime it supports, Linux multi-runtime support just falls out.

<comment> vezzy-fnord nspawn containers are yet not really used as a platform target, and currently Docker is leading. Though nspawn has since acquired some notion of the Docker image format, and even in light of the OCI standard, I do not foresee it becoming a primary container solution in its present form. rkt being nspawn- based may or may not take hold.

Indeed, systemd is as of yet not comprehensive enough to be a POSIX-parity target. There is no "systemd Linux" as such, it's systemd/GNU/Linux or Linux with the GNU and systemd suite. Android is a top-down integrated system on the other hand and does not linearly track GNU/Linux.

As of yet, there is nothing like system call emulation or similar in nspawn to have branded zones.

Nor does Red Hat's present actions imply something like this. The GNOME project, affiliated with Red Hat, is working on a poor reinvention of Nix called xdg-app to enable the "app frameworks and runtimes" design that Lennart Poettering wrote about in "Revisiting How We Put Together Linux Systems," but that too is firmly specific to GNU/Linux as the host.

Red Hat is also leading a container OS called Project Atomic, however nothing like branded zones is seen there as a goal, either. Instead, they've made a simple meta-framework for running various _Linux_ container images over several orchestration frameworks and PaaS, called Nulecule. It's firmly a layer over Docker and Kubernetes, however, so it is limited to that.

 _A Linux "ReactOS Runtime" equivalent to Windows's "POSIX Runtime."_

That would be quite a feat in of itself, systemd or branded zones aside. ReactOS isn't Wine, but even with Wine it would be a sizable integration effort.

 _If that init daemon is effectively a container-manager that understands how to instantiate and manage different "branded containers" for each runtime it supports, Linux multi-runtime support just falls out._

The init daemon is not a container manager in this case, but an object- oriented resource management with a transactional job scheduler and some limited execution environment modification that work with namespaces and cgroups, but the container framework is outside. As it should be. An init daemon as your container manager sounds dreadful and horrifying, though I hear RancherOS boots from Docker as PID1...

But as there is no such init daemon yet nor anything like branded zones, I still have to say you're a dreamer. This _might_ be a long-term strategy, but with acts like Project Atomic it really doesn't look like it. I still think it's more middleware than Core runtime. I pray it is...

<comment><USER8> Note that I wasn't suggesting that systemd is _itself_ an attempt at an "init daemon with branded zones"—instead, it's a particular runtime that will increasingly differentiate itself from POSIX Linux, and I believe that that _friction_ will eventually cause developers to want to _create_ an init daemon with branded zones to supercede systemd, where emulating systemd-as-it-stands (for apps built to expect it) would be one of that multi-runtime system's goals. With the POSIX people and the bigcorp Linux providers pulling in opposite directions, a multi-runtime Linux would be the only Nash equilibrium. (Though it'd need someone like FreeDesktop to suggest it and start a working group for it, because neither "side" would care about it on their own.)

That may not be the way things play out, certainly. Things might just diverge and stay that way, if nobody cares enough to change things. But there are unknown unknowns that can give things a very hard shove in that direction. For example: imagine that systemd decides to integrate deeply with GNOME, to the point that you now have an integrated "systemd+glib+GNOME platform" with a unified API, the way that OSX is a "launchd+CoreFoundation+Cocoa platform". _That_ would be an extremely divergent target in the Linux ecosystem introducing a lot of friction to everyone else's development efforts—similar to the early stages of the Android project—and it would get a lot of people's hackles up. But it's not out of the realm of possibility for RedHat and Debian to agree on something like that.

<comment> vezzy-fnord _instead, it 's a particular runtime that will increasingly differentiate itself from POSIX Linux_

Ah, correct.

 _I believe that that friction will eventually cause developers to want to create an init daemon with branded zones to supercede systemd, where emulating systemd-as-it-stands (for apps built to expect it) would be one of that multi- runtime system 's goals. With the POSIX people and the bigcorp Linux providers pulling in opposite directions, a multi-runtime Linux would be the only Nash equilibrium. (Though it'd need someone like FreeDesktop to suggest it and start a working group for it, because neither "side" would care about it on their own.)_

It's a hypothesis, then. Plausible, certainly. I assume you're presenting this as a probable way out of the portability concerns from GNU/Linux becoming more of a closed environment to the point of instigating a massive developer schism? This still raises the question of why it would need to reside in the init daemon, but I suppose crass architectural decisions are not a stranger to Linux developers.

 _For example: imagine that systemd decides to integrate deeply with GNOME, to the point that you now have an integrated "systemd+glib+GNOME platform" with a unified API_

Such a decision might just backfire quite tremendously and prove to be systemd's overturning, though it does sound like a similar integration is very much in the spirit of the project. Then, yes, radical surgery would occur, but it may not be quite as dramatic as branded zones and simply an abandonment of systemd and related Freedesktop-ware.

 _RedHat and Debian to agree_

Debian aren't really that large of a player, more of a passive target platform. Being a non-affiliated foundation, they're generally steered into things by whoever is on the committee and seldom ever lead themselves. So, no agreement strictly necessary.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> Read it through but hard to grasp all the details. One thing I'm sure is that for non-systemd "legacy" init systems I know what I'm doing, for systemd I most of the time don't know what's going on there, it's still like a blind art after using it for a while, overly abstracted might be one reason for that.

<comment><USER10> I think part of the reason for this may be that some types of systems lend themselves to exploration and gradual learning through use, while others, being very tightly coupled, generally require a larger upfront investment in learning how they function as a whole, since any one subcomponent is only marginally useful without the rest.

This is largely how I remembered studying for Microsoft certification back in the day to be. There was only so much you could easily learn from playing around with the system. At some point, well written technical documentation was, if not required, _extremely helpful_ in making connections between how systems worked together, or explaining exactly what a component was or was not capable of. I don't think this is inherently worse or better, just _different_ , and with its own advantages and disadvantages. For example, people who went through this training seemed to have a very high base-level of understanding of how to configure these systems, what they could or could not do, and how to troubleshoot them, but not necessarily how they _functioned_ in every case. Conversely, the loosely coupled individual components of traditional UNIX service handlers (sysvinit, inetd) promoted people learning how the actually work, and what they do, but with scattered and not entirely comprehensive documentation on how they worked. The average person responsible for using these will have vastly different levels of skill and knowledge about them, but with that comes how it actually functions, because that's needed to troubleshoot problems, since it's so exposed.

The feeling I get mostly when reading about systemd is that it's very much in the vein of MS products. Like I said, that isn't necessarily a bad thing, but it is very different than a lot of UNIX admins are used to.

<comment><USER11> >Conversely, the loosely coupled individual components of traditional UNIX service handlers (sysvinit, inetd) promoted people learning how the actually work, and what they do, but with scattered and not entirely comprehensive documentation on how they worked.

Can you tease out this distinction a bit more? It reads like you are saying, due to their nature, people learned how to use Unix service handlers, without a good understanding of how they were designed. It could also mean that people learned how Unix service handlers worked by using them, through trial and error, because the documentation was scattered and not entirely comprehensive.

<comment><USER10> The second interpretation. The documentation is not entirely comprehensive because interface between components is not only a grey area of accountability, but also continuously changing. When the system is composed of multiple distinct sub-programs with their own development teams, update cycles and planned feature sets, what is possible at any one point in time is largely a matter of which versions of those sub-programs you are using, as well as what the root program (e.g.sysvinit) allows in it's version. There is no one set of features, nor one set of bugs, nor one set of directions. The best you can get in this regard is whatever your distro put together for the combination they chose, or alternatively to read the docs for every sub- program and intuit what new capabilities may exist in how they interact.

I think the clearest example of this would be everything that makes up the core Linux environment, compared OpenBSD. OpenBSD has amazing documentation, and the systems are all designed to work with eachother because they were all designed and developed as a working whole. Linux, on the other hand, often exposes interfaces through the kernel, and some other set of developers (or multiple set of developers) create systems to interface with it. As such, the documentation can be haphazard or near non-existent. That's not to say the Linux methodology doesn't have it's advantages. For example, I think there's a better feedback loop in Linux in determining what is needed from the kernel, and what can be done from it in userspace. It's a more market-like system, and sees rapid advancement.

<comment><USER11> Thank you!

</comment></comment></comment></comment></comment><comment><USER12> Very interesting article, refreshing as well to see a criticism of systemd based on technical details. Unlike most other criticisms of systemd, this author does not resort to personal attacks and insults against Lennart Poettering or fear-mongering appeals to tradition. I still think systemd is an improvement over the previous sysvinit, but the author seems to have found some real conceptual flaws with systemd that are somewhat unsettling.

I hope this article can start some actually useful conversations about init systems that do not fall into the pitfalls of FUD and negativity that seem to have surrounded this topic in the past.

<comment><USER13> My explanation for the lack of technical (rather than political) articles examining systemd is that it's very difficult to wrap your mind around because it carries so many ideas. A quote from an article linked within the linked article:

> All of the above definitions are technically correct. None of them are > complete.

> It is thus that debates are doomed to go nowhere, as no one can agree on a > single definition of systemd, and the developers have been largely unhelpful > in that matter. Vagueness can be a strength, in that it allows for people to > divide and conquer, but it is also unproductive and leads to unnecessary > bloodshed.

Slowly, surely, people will begin to understand what systemd is. Only then will people be able to figure out whether they want it or not.

</comment></comment><comment><USER14> The major problem right now if one decides to use a systemd alternative is that most packages are shipping with systemd unit files, but will obviously lack support for other init implementations.

I like dmd, implemented in Scheme, which is part of the GuixSD distribution.

<comment><USER15> 1\. The solution to this is to write a translator from systemd units (those of .service type) to the desired kind of unit. I haven't looked into dmd, but this should be possible at least for the majority of units.

2\. This was going to be a problem in any case i.e. had most distros switched to openRC one would still have been faced with the task of producing a suitable collection of service configuration files for dmd and then adapting them to dmd's specfic featureset.

<comment><USER16> Yeah, well, if it's to go from unit files to Guile code, that's not what I'd consider an improvement. Having configuration files for use cases where the requirements are simple enough is nice.

Of course, when they're not, you end up with an abomination like MSBuild, but that's not the case here.

<comment><USER15> True, but if that is the case you won't be running dmd in the first place. And automated conversion/runtime translation will be a possibility even with different service managers, e.g. systemXVI.

I'm not really convinced by the second, in that I believe that nothing in the article proves that service management is something too complex to be done via configuration files.

</comment></comment></comment></comment><comment><USER17> The article that lies herein, was, in a manner of speaking (but not literally speaking - 'in a manner of speaking' is just a phrase), so overly verbose in many places that it drowns the reader in verbal diarrhea as thick as pea soup. That is to say, I, as a reader, felt that the author (of this article) could have made these points in a more reachable, succinct fashion.

<comment> vezzy-fnord I would, as a general principle, be cautious (in the sense of not being hasty) to dismiss in a manner reminiscent of Parkinson's law of triviality (insofar as the latter pertains to the bikeshed principle) the literary style of the article in question, since the common interpretation of excessive verbosity is flawed, in this case being seldom more than a preference with little in the way of any concrete justification that would be of direct relevance to the topic at hand, the topic being an architectural treatise. Any reservations about succinctness, or a perceived lack thereof, must therefore be qualified and left over for further analysis to draw proper and conceptually robust remarks given the evidence presented.

(Yes, the writing style is somewhat obtuse. You'll have to excuse that. The topic itself is obscure, so I suppose it fits.)

</comment></comment><comment><USER18> My response from another discussion on this article:

Much of these deficiencies boil down to the design criteria for systemd, which is basically to avoid forcing the system administrator to perform dependency graph resolution. This was a direct response to Upstart, which pushed this work to the administrator (or distro packager) by design.

Thus the declarative syntax for unit files, which do not explicitly control dependency resolution, parallelization, or startup order, which is simply not necessary or cared about for 99% of use cases. The 1% case there is for situations where you should not use a prebaked init system in the first place, such as when you absolutely need deterministic boots, i.e. in an embedded or realtime context.

> In systemd, however, the execution state is not based on explicit chain > loading, but on serializing unit file options to a private ExecContext > structure overlayed into Unit objects and set for service unit types.

This is not a problem with systemd's design. This is a problem with software written without the assumption of reliable process supervision, which is a reasonable assumption given the history of UNIX daemonization. systemd takes the (apparently controversial) stance that processes should not manage their own execution environment, or that of their dependencies. It provides numerous prebuilt hacks to make things work for broken software, but nobody should expect systemd to design for 100% of situations where processes do Weird Shit™.

And yes, I'm including "delegated service restarters" under this umbrella. The need for something like this absolutely smells of poor design. It also flies in the face of the author's preference for deterministic execution behavior, because by definition your process will execute differently depending on if it was started or restarted.

The link to "systemd house of horrors" [1] only proves this point.

> A supervisor insisting on programmatic accommodations from service writers > is not the most desirable state of affairs. A rarely discussed alternative > to the two common approaches is, again as with startup speed optimization > minus parallelism, checkpoint/restore: checkpoint a process image from a > point where initialization is known to be complete and overlay it on > startup, using a tool such as DMTCP or CRIU.

I don't know about anyone else, but this sounds like an absolute nightmare. For one, it relies on the service exposing enough information to be checkpointed at the right time; at which point, you might as well call sd_notify or hook in a script to do it for you, so this is not a reduction in requiring programmatic accommodations from service writers. Two, process- snapshotting is not a panacea, and is a nightmare to debug when it breaks your expectations; you'll need to test all sorts of process state, including open pipes/fds/sockets, signals, system calls (!), shmem buffers, etc. At which point it's best to just let the service notify you directly when the process is done. Which is a facility that systemd provides.

> Of course we also have the issue of circular dependencies in the systemd > architecture itself. We have the init, process manager, process supervisor, > cgroup writer, local service tools, the Unit object structure (which might > benefit from being made a protocol), timers, mounts, automounts and swaps > all in the same module with ill-defined boundaries.

A worthy criticism. I would also like to see a bit more modularization from systemd, but I can also see the point of systemd developers not expending effort to do so. For one, it would drastically increase the number of contexts for testing (i.e. can we verify the service manager works without the cgroups module loaded, etc). I think systemd is at the point where it's rock-solid for all of its use cases, and a pluggable/composable architecture would have delayed this situation for not very much benefit.

In short, systemd imposes its opinions about process init and supervision. Whether or not this is reasonable is up for debate. However, it is very useful to have both a strong opinion and a solid implementation of this opinion.

IMO, the idealized world that systemd supports (i.e. non-forking services, initialization notification, strict execution environment and dependency declaration, centralized and opaque init/dependency- resolution/parallelization) is a vast improvement over the status quo and actually improves the lives of service writers (by removing an entire class of responsibilities), so I think it's a reasonable opinion and thus I think systemd has a reasonable design and architecture.

There are many known technical deficiencies with the implementation that are mostly being worked on, but I'm not seeing anything damning of the whole idea.

[1] [http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/sy...](http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/systemd- house-of-horror/)

<comment> vezzy-fnord It appears you received a response here: [https://www.reddit.com/r/linux/comments/3od733/structural_an...](https://www.reddit.com/r/linux/comments/3od733/structural_and_semantic_deficiencies_in_the/cvwf201)

You do indeed misunderstand the point about execution composition. Chain loading is indeed configuration-related, not service-related. It's a much more flexible and visible method than internal structures. Delegated restarters then can be deterministic as they reuse process management.

C/R solutions don't require special integration, that's the point. They work on unmodified processes, unlike readiness notification. This is I think a reasonable trade-off for the potential decrease in reliability, as the feature comes with a double advantage of startup optimization and then readiness notification is generally not _that_ critical, in light of regular Unix process semantics. C/R is also constantly improving and reasonably stable for base use cases these days.

<comment><USER6> You seem to dislike the idea of asking for notification support in the daemons themselves. Granted, it's impractical, but it's the only way to avoid race conditions which are rare for most services, but _very real_ for a few programs that may take ages to get ready. How would you address the problem of slow starters _without support_ from the daemons, while keeping a "the dependency graph is always respected" invariant?

<comment> vezzy-fnord Right, if the problem is slow starters (say, RabbitMQ), then restoring from an image where initialization has already been complete would resolve it. For more complicated cases where there is a race with acquiring some resource, then a readiness notification mechanism could be supported as a niche feature. I'm still not convinced of it in the general case.

<comment><USER6> But how would you go with restoring such an image? It won't be instant, there will still be a window where the service manager is continuing its work but the image hasn't been restored yet. One way or the other, you need a synchronization mechanism, so how would you design such a mechanism without support from the daemon?

You don't need a complicated case to have a race with acquiring some resource. A server opening a socket is providing a resource; clients connecting to that socket are using it. Clients should not be started before the server is listening to the socket. How would you solve this case - are you advocating socket activation by the service manager? ;)

</comment></comment></comment></comment></comment><comment><USER19> Thank you for doing this - I'm going to read this with a lot of interest :-)

<comment> vezzy-fnord Do return with commentary.

</comment></comment><comment><USER20> tl;dr: the author argues that:

* systemd is more similar to a traditional job scheduler than a traditional init system.

* systemd should not parse text files in pid1 because it might be a security risk.

* Because systemd handles order dependencies, it is sometimes susceptible to "ordering cycles" where a clear ordering cannot be established, "dependency loops" where jobs are continuously dropped and requeued, and race conditions if dependencies are accidentally underspecified.

* "Imbalance between promoting laziness or eagerness": launchd, a predecessor of systemd, operated purely "lazily," launching services only when other services needed them. systemd also supports launching services eagerly, which the author argues is more complex.

* Checkpointing a process image could have been used instead of systemd's readiness notification mechanism (?)

* Systemd does not have a plugin system

* Journald is criticized because it does too much

I would counter that:

* I agree that systemd is similar to a job scheduler, but I don't necessarily believe that that is a bad thing. Manual scheduling of jobs for startup and shutdown like we did in the rc.sh days is complex and error-prone.

* The text files systemd is parsing are owned by root, so even if the parser is exploitable, only root can take advantage? Seems like a weak criticism.

* I agree that dependency ordering makes things more complex. But upstart and launchd have the same issues. It seems to be a tradeoff worth making.

* I don't understand the argument against supporting both laziness and eagerness in dependencies. It seems like some things are naturally modelled eagerly, like needing to perform a bunch of somewhat unrelated actions to suspend the system. And some things are better handled lazily, like setting up a FUSE filesystem when a USB stick is inserted that needs it. Shouldn't we use the model that makes the most sense?

* I don't think checkpointing a process image can ever really replace having a notification system. Even if the checkpointing code could be make bulletproof somehow, some processes deal with state in the external world, or with hardware, that makes checkpointing infeasible.

* Regarding a plugin system: Systemd has the ability to run shell scripts, which can be useful in filling in gaps in functionality. I don't think a more complicated plugin system would be a good idea since it would add a lot of complexity (and potentially instability.)

* I didn't understand the criticism of journald, maybe someone can elaborate. The author presented some alternate approaches but I missed why these were better (other than the handwavey argument that journald was "a bottleneck")

I enjoyed reading this, and it's nice to see some more reasoned criticism of systemd. I feel like the prose got a little bit purple at times. We had to spend 10 paragraphs "descend[ing] into an inferno with the same dead horse talking points", "culminating into some rather heterodox conclusions", and "progressively introducing and elucidating on concepts, applying some a priori reasoning at times, and backtracking to derive conclusions or reiterate on prior stated knowledge" before we even saw a word of argument! This needed an editor...

<comment> vezzy-fnord Actually the job scheduler is one aspect, complimentary to the object-oriented resource management which I made quite clear was the main point.

Regarding file ownership, not necessarily. If you run it as a session-based init, they'd be owned by the user it serves as an agent to. (see dirs: [http://www.freedesktop.org/software/systemd/man/systemd.unit...](http://www.freedesktop.org/software/systemd/man/systemd.unit.html))

Upstart and launchd do not have the same issues. Upstart doesn't have dependencies, but named preconditions and postconditions. launchd has neither that nor dependencies. They have many problems of their own, but different ones.

My point regarding laziness v eagerness is the way systemd implements multi- paradigm introduces undesirable externalities from both approaches, as I said: "In systemd, one must endure both the non-determinism from aggressive parallelism via dependency information, the object network needing the transaction and job metaphors, all while having its lazy features be thoroughly restricted to its internal Manager and not a flexible toolkit such as UCSPI."

I'd assume the limited use of C/R to simply cut down initialization times _is_ feasible: [http://criu.org/Usage_scenarios#Slow- boot_services_speed_up](http://criu.org/Usage_scenarios#Slow- boot_services_speed_up)

Running shell scripts doesn't mean much when the underlying system has a large surface and ill-defined module boundaries. A plugin system most certainly does not have to be complicated: look at finit and initng for examples.

What about journald did you not understand? You lose control over the separate stages of collection, rotation and processing. Particularly having custom postprocessing is important for interoperability with foreign services that might expect certain format constraints. Having the logs be per-service instead of globally snarfed is also saner.

Yes, I did go overboard on the reservations. It's a flame war topic, so I felt it necessary as a precautionary measure.

<comment><USER20> _Regarding file ownership, not necessarily. If you run it as a session-based init, they 'd be owned by the user it serves as an agent to._

Interesting. I didn't realize systemd had a "user mode" where it doesn't run as pid 1. Does systemd run as root (or root equivalent) in this mode? I wasn't able to find this from a quick skim of the information available online.

 _Upstart and launchd do not have the same issues. Upstart doesn 't have dependencies, but named preconditions and postconditions. launchd has neither that nor dependencies. They have many problems of their own, but different ones._

Sorry, but I have to disagree. The terminology may be different, but ultimately upstart's "start on" directive establishes a dependency. Once you have a dependency you can have circular dependencies. And once you rely on the init systems to establish ordering rather than having everything just start up in a deterministic order in simple single-threaded script, you can have race conditions where things happen to work on one machine but not another. I'm not as familiar with launchd; maybe its socket activation makes the circular ordering harder to hit, but fundamentally there can always be races.

 _I 'd assume the limited use of C/R to simply cut down initialization times is feasible: [http://criu.org/Usage_scenarios#Slow- boot_services_speed_up](http://criu.org/Usage_scenarios#Slow- boot_services_speed_up) _

Perhaps for some processes this is feasible, but nothing on that site addresses my objections above: "some processes deal with state in the external world, or with hardware, that makes checkpointing infeasible." You will always need a notification mechanism even if you have checkpoint and restore. And why have the complexity of checkpointing if you have a working notification system?

 _Running shell scripts doesn 't mean much when the underlying system has a large surface and ill-defined module boundaries. A plugin system most certainly does not have to be complicated: look at finit and initng for examples._

From a user's point of view, I do not want a plugin system. I want systemd to work the same on SUSE vs. debian vs. Red Hat. It is usually better to put things into the base system than to have them live in a half-supported state outside the tree. That's why web browsers are killing plugins: in order to be useful, they have to reach deep into the guts of the program, which is not a place I want 3rd party code.

 _What about journald did you not understand? You lose control over the separate stages of collection, rotation and processing. Particularly having custom postprocessing is important for interoperability with foreign services that might expect certain format constraints._

I don't understand why I would want separate daemons for collection and rotation, at least for dmsg, kmsg, and syslog. logrotate always had a bunch of race conditions; it was never a great design.

I agree that journald doesn't do custom processing. However, I'm a little unsure whether I would really want it to, or whether I'd be better off running my own syslogd implementation at that point. That is still an option, even when you are using the rest of systemd.

 _Having the logs be per-service instead of globally snarfed is also saner._

You can always set up important services to log to a file rather than to syslog.

My real criticism of systemd is that they made no effort at portability outside of Linux (and I don't buy their arguments about why it's too hard.) It would have been a lot less controversial if they had at least pointed out how the BSDs could add support (even if they themselves didn't do the work).

<comment> vezzy-fnord _Once you have a dependency you can have circular dependencies._

Then the question becomes whether an architecture obscures or exacerbates them, and also how it deals with them. The systemd object model does not fare well, as discussed.

(Also compare a system like s6-rc or serel where dependency graphs are compiled from manifests and consistency checked before being applied. The graph is persistent configuration and not an ephemeral artifact.)

 _And once you rely on the init systems to establish ordering rather than having everything just start up in a deterministic order in simple single- threaded script, you can have race conditions where things happen to work on one machine but not another._

Systems having ordering dependencies does not inherently affect their determinism, the startup discipline (sequential v. parallel) primarily does, and then there are different ways to tackle parallelism. systemd's execution model and transactional object system exacerbate non-determinism in ordering results.

 _I don 't understand why I would want separate daemons for collection and rotation, at least for dmsg, kmsg, and syslog. logrotate always had a bunch of race conditions; it was never a great design._

They're not separate daemons.

 _My real criticism of systemd is that they made no effort at portability outside of Linux (and I don 't buy their arguments about why it's too hard.)_

That's a _horrible_ criticism of systemd. Any cursory examination of systemd will quickly reveal the task is next to insurmountable without doing what amounts to a full reimplementation.

<comment><USER20> _Then the question becomes whether an architecture obscures or exacerbates them, and also how it deals with them. The systemd object model does not fare well, as discussed._

What specific aspects of the systemd dependency model do you feel exacerbate race conditions and circular dependencies?

 _(Also compare a system like s6-rc or serel where dependency graphs are compiled from manifests and consistency checked before being applied. The graph is persistent configuration and not an ephemeral artifact.)_

I have seen people discussing systemd error messages about circular dependencies online. It seems that systemd does perform some checking, although I haven't found documentation on what checks it does (I probably missed it). I'm not sure how the dependency graph could be persistent configuration because hardware can be hotplugged these days. For example, are you going to create nodes in the graph for all the hard drives that could possibly exist over the lifetime of the machine?

 _Systems having ordering dependencies does not inherently affect their determinism, the startup discipline (sequential v. parallel) primarily does, and then there are different ways to tackle parallelism. systemd 's execution model and transactional object system exacerbate non-determinism in ordering results._

If the sequential vs. parallel startup discipline "primarily affects determinism" won't all these systems have the same issues with nondeterminism? I don't think anyone is seriously proposing starting up services on boot sequentially in 2015. Or even when a USB stick is inserted. Systems have 12 cores now and people can be very sensitive to boot times.

 _[lack of portability is] a horrible criticism of systemd. Any cursory examination of systemd will quickly reveal the task is next to insurmountable without doing what amounts to a full reimplementation._

I understand that systemd relies heavily on cgroups and other Linux-specific features for process isolation. And the BSDs are also unlikely to accept GPLed code. However, I feel like they could have at least discussed some way of reducing fragmentation here. As it is, I would be amazed if at least one or two of the BSDs didn't implement a similar init system to systemd in the next few years. And as it is, they probably won't share any code or concepts, unfortunately.

<comment> vezzy-fnord _What specific aspects of the systemd dependency model do you feel exacerbate race conditions and circular dependencies?_

Well, the article discusses all of that...

(Speaking of which, JdeBP helpfully linked to Upstart's pre/postconditions not being a proper dependency model, per the developers' testimony, and as I asserted.)

 _I 'm not sure how the dependency graph could be persistent configuration because hardware can be hotplugged these days._

The idea of device units is not well borne out, as I mentioned. You don't need to create an object type out of a udev handle to order after devices or trigger hotplug events (the device node manager itself does fine, of course in the case of systemd, udev _is_ part of the project, so...)

 _If the sequential vs. parallel startup discipline "primarily affects determinism" won't all these systems have the same issues with nondeterminism?_

Not necessarily. It could either be more manageable if there wasn't so much indirection, see above (I'm afraid you just didn't get the article, and your original summary certainly implied as much). Additionally, one could forfeit being _aggressively_ parallel for a more conservative approach like in OpenRC, s6-rc or even Upstart.

* I don't think anyone is seriously proposing starting up services on boot sequentially in 2015. Or even when a USB stick is inserted. Systems have 12 cores now and people can be very sensitive to boot times.*

Parallelism is but a small part of the complicated topic of boot time optimization. And yes, sequential strategies can still be really fast. Much of the speed increase in modern systems is not from parallelism, but forfeiting the overhead of the shell interpreter in favor of direct execution. Checkpointing can do far greater wonders for your boot times than parallelism can, anyhow.

 _I understand that systemd relies heavily on cgroups and other Linux-specific features for process isolation._

That's only scratching the surface of Linux-specific features in systemd.

 _As it is, I would be amazed if at least one or two of the BSDs didn 't implement a similar init system to systemd in the next few years. And as it is, they probably won't share any code or concepts, unfortunately._

What will come of the BSDs is uncertain. launchd will be used in iXsystems products like PC-BSD and FreeNAS, and there's not much "code sharing" you could do there - it's already a complete product being ported.

<comment><USER20> _Parallelism is but a small part of the complicated topic of boot time optimization. And yes, sequential strategies can still be really fast. Much of the speed increase in modern systems is not from parallelism, but forfeiting the overhead of the shell interpreter in favor of direct execution. Checkpointing can do far greater wonders for your boot times than parallelism can, anyhow._

No. I have worked on embedded Linux systems where we have carefully charted boot time. We were able to get order-of-magnitude improvements by parallelizing service startups during boot. Parallelism is far from "a small part"\-- it is the only important part. The overhead of the shell is negligible even on the slowest systems. Checkpointing is not an option for most things (I have mentioned this several times) because most of the startup processes initialize the hardware directly or indirectly, and hardware can't be checkpointed.

 _The idea of device units is not well borne out, as I mentioned. You don 't need to create an object type out of a udev handle to order after devices or trigger hotplug events (the device node manager itself does fine, of course in the case of systemd, udev is part of the project, so...)_

The important question is whether there is more or less complexity in systemd than a system with a separate device node manager. I am not convinced that systemd is more complex than the alternative. If the device node manager was separate, you would have to debug the interaction between the two processes. Having dealt with debugging d-bus interactions on Linux before, it doesn't sound fun.

I'm not convinced that a static dependency graph is even feasible in a world where users can reconfigure and install new services, hotplug hardware, and join and leave wireless networks.

</comment></comment></comment></comment></comment></comment></comment><comment><USER21> My take-away is the following:

* systemd tries to solve a complicated problem

* systemd has a lot of complexity

* some of that complexity seems unnecessary

* parsing is hard (see DJB). Parsing (as systemd does) in PID 1 is bad. Complex systems in PID 1 is bad.

* systemd manages states and dependencies (this is good)

* some of the states are only internal, and can't be serialized to storage. The result is indeterminism and race conditions. (This is bad)

* as a result of the above point, systemd behavior is hard to debug, and hard to understand

* another result (not stated explicitly) is that systemd allows for (and apparently can't detect) ordering cycles and dependency cycles.

* The configuration files are simple for simple tasks (good), but not complex enough for complex tasks (bad)

... etc.

i.e. systemd is an engineering solution. It pretty much works, but it has inherent design / implementation issues which cause real-world problems.

<comment><USER22> This article pretty conclusively convinced me that systemd is actually good, if these are the strongest arguments against it. Most of the dependency- related stuff (process order related, cyclicstuff) seems eminently fixable. I found his arguments about dependency management having to be linearizable highly unconvincing.

<comment><USER23> > I found his arguments about dependency management having to be linearizable > highly unconvincing.

What? Can you expand on this? To me it rang as a well-reasoned, and pretty devastating, critique.

</comment></comment></comment><comment><USER24> "a controversial topic"

My observation: The "discussion" never seems to focus on systemd. Instead it usually turns to comments on the former sysvinit system or other init systems.

That avoidance is perhaps something to ponder. And maybe it's why this author felt the urge to put some focus on systemd itself.

Love the quote from djb. In sum, the best interface is no interface. Parsing amounts to high margin for error and often an incredible nuisance.

In another thread today I wrote about the command line interface and "expecting a reponse". Truthfully, djb's utilities that simply return an exit value and no output are the best ones I have ever used.

(It seems djb himself is a systemd user. Not sure what if anything that means.)

Imagine if the information dissiminated via www was as easy to "parse" as text0. Writing a simple "web browser" might be easy enough that programmers would not need to be paid to do it.

<comment><USER2> > _The "discussion" never seems to focus on systemd_

All the more reason to try making this one better.

(We detached this comment from [https://news.ycombinator.com/item?id=10370446](https://news.ycombinator.com/item?id=10370446) and marked it off-topic.)</comment></comment>


The Burger King and Jeep Hacker Is Probably This DJ From New England  - <AUTHOR> http://gizmodo.com/5985353

<comment><USER2> What happened to the entertaining, informative giant that used to be the Gizmodo/Lifehacker/Kotaku network? They've faded into a bunch of pageview garnering alarmists. Someone got me thinking like this, I don't remember who, but the more I look at it, the more it seems like they're just capitalizing on current events to garner page views instead of trying to... Oh, I don't know... Do something useful.</comment>


Why You Should Talk To Investors (Even If You're Not Raising Money) - <AUTHOR> http://www.erica.biz/2012/talk-to-investors/

<comment><USER2> it is the duty of every good person to help each others.if you make me your subcribers raise money for me in my name which i will retun to you with thanks.if you are my friend it is also your roll to a friend and which will retuned to you.i promised you money earn will be utalise for welfare of the people.yer or no but please give me reply.real friendship begin in this way.

</comment><comment><USER2> i only deal and talk about why the good persons are not dominating the world inspite of that every one want to be good.it is true and we have to create logics to help in domination of good persons we should try every where.so we should talk every good person inspite of that he is not raising money.

</comment><comment><USER3> Liked the video pitch. Nice how the discussions with investors crystalized your thinking. Great post.

</comment><comment><USER4> Really clever stuf Erica... always love your posts

</comment><comment><USER5> Awesome, awesome stuff. Oh yeah.

</comment><comment><USER2> we also want to give the proof that we are unselfish.which definatly attracts masses.

</comment><comment><USER6> yes - because you never want to start talking to them when you need it!</comment>


Reactive Cloud Actors: No-nonsense micro-services - <AUTHOR> http://byterot.blogspot.com/2014/04/reactive-cloud-actors-no-nonsense-microservice-beehive-restful-evolvable-web-events-orleans-framework.html <comment><USER2> My bullshit detector was tripped by the number of buzzwords.

What does this actually solve versus say traditional MQ (EIP stuff), service bus systems and normal SOA?

I've dealt with NServiceBus extensively and really big scale .Net stuff and if it's anything remotely close to that architecture, I'd rather hang myself than use it again.

<comment><AUTHOR> Yes, so many buzzwords together - I admit.

The difference is replacing Commnad/Query with events. Replacing Imperative with Reactive.

Did you read the whole post or you were put off by the buzzwords?

<comment><USER2> Perhaps it's because it is Sunday but I phased out after the first couple of paragraphs. I've read it now and am still unaware of why this is fundamentally better.

Command/Query pattern is an imeplementation concern. The architectural pattern that backs it up isn't necessarily related (for example event sourcing). At that point it isn't imperative so why is this different? It just appears to be fragmenting lots of concerns into even smaller ones which are vastly more complicated to debug and manage.

The bold highlighted text of: _" in a reactive actor system, each actor only knows about its own Step and what itself does and has no clue about the next steps or the rest of the system"_

Isn't that the same for a message channel but without a transactional guarantee? (going back a decade or more to EIP).

<comment><AUTHOR> No it is not the same. First of all, CQRS is applying CQS to the architecture, so it is an architectural decision. When you use commands, you know about the interface of the endpoint receiving your message. In case of an event, you do not have to know anything - and this is the whole premise of this post. In the article to come, I have explained the differences more with regard to EIP as well. A half-draft available at [https://gist.github.com/aliostad/55b652e44dfe87d44444](https://gist.github.com/aliostad/55b652e44dfe87d44444)

<comment><USER2> Thanks. I'm sceptical (having been through several similar systems and ripped them out) but I will read your draft later.

</comment></comment></comment></comment></comment><comment><USER3> He's certainly on to something. As he pointed out, it's been around since about '73, and many people are happy with the design of such systems. Amazon even has an actual for-real implementation of one (see Simple Workflow), he should probably try it out before rolling his own.

It really is an elegant way to do things from what I can tell. It forces you to keep your state in one, carefully managed place (hairball), rather than sprinkled around god-knows where, being poked and prodded by god knows what....

IMHO, There should be more talk about the design of actor based systems than the glut of OO yammering we are all exposed to.

</comment><comment><USER4> Another case of reinventing the wheel and calling it something new. This is just a reimplementation of what used to be called the "blackboard architecture".

[http://en.wikipedia.org/wiki/Blackboard_system](http://en.wikipedia.org/wiki/Blackboard_system)

I don't see how broadcast is more "reactive" than singlecast. And the solution to tight coupling in actor systems has already been thought of - it is a message broker. This is just an actor system where every actor shares a single message broker.

Its cool that you built your own system for actors, and I certainly wouldn't discourage people from reimplementing something for the sake of learning, but every criticism you have of traditional actor systems is inaccurate. Strangely enough, you criticize Orleans in particular, which is especially good about tackling the coupling problem by making actor references virtual.

"Perhaps the central problem we face in all of computer science is how we are to get to the situation where we build on top of the work of others rather than redoing so much of it in a trivially different way."

-Richard Hamming

</comment><comment><USER5> The big problem with this kind of stuff is how debug them. Also, that despite the intention, all is really coupled: Don't make sense to cancel a order before create it, so the order of the actions are important.

The benefit is (only ?) easy of scaling, IMHO. How solve this? I understand that GO keep the ordering of the calls and is not harder than doing imperative...

<comment><AUTHOR> Well, first of all, debugging while developing an actor is actually easier: you just have to debug an actor in isolation and make sure it does the right thing and sends the right event in response to the incoming event.

In terms of debugging issues when in live, you have to rely on tracing and let something such as Logstash handle search, etc.

The ordering of the workflow is preserved. As I explained, FraudCheck is only ever done after an order's payment has been authorised.

Hope this makes sense.

<comment><USER6> _Well, first of all, debugging while developing an actor is actually easier: you just have to debug an actor in isolation and make sure it does the right thing and sends the right event in response to the incoming event._

No. No, this is terrifyingly incorrect. Making each component do the right thing in isolation is easy. Most of the difficulty in systems - in developing, testing, and debugging them - arises from what the components do when they're together, because it turns out your ideas about how they need to behave in isolation were mistaken.

<comment><USER5> Exactly. Sometimes is necessary to step into the call of another function/actor, see what is is doing, note is necessary to step into more into a deeper function, etc.

Or stop when a certain value arrive, and inspect the code. If this could be done with actor/csp, then is good idea to jump onto it.

</comment></comment></comment></comment><comment><USER7> Isn't this what [http://akka.io](http://akka.io) does?

<comment><AUTHOR> Yes, but they do them differently. BeeHive is much more opinionated about the message (must be event and has an enveloper) and the Basic Data Structures.

Also BeeHive avoids stateful actors. State is always persistent in HA queues or Basic Data Structure stores.

<comment><USER8> I thought Rails had taught us that extremely opinionated infrastructures are doomed to at best be influential, but not widely used.

Sometimes actors have to be stateful. Sometimes they should be the interface to large, complex computations that take 20 minutes to run.

The market is choke full of techs that solve very narrow cloud problems. What we need is something that isn't so terribly narrow. Microservices is very, very narrow. Even people working for Fred George admit to that.

<comment><USER9> I'm having a hard time parsing this—Rails is something I would consider to be both highly opinionated and widely used, but its unclear where your statement falls. I'm not crazy about some (most?) of the opinions, but that's a matter of some debate.

> Microservices is very, very narrow. Even people working for Fred George > admit to that.

Its hard to make this statement without first agreeing to a definition of microservices. I'm not convinced there's wide agreement to such a definition yet.

<comment><USER10> Some have attempted this, for real or for giggles.

[http://www.russmiles.com/1/post/2014/04/the-microservices- ma...](http://www.russmiles.com/1/post/2014/04/the-microservices-manifestno- dutch-edition.html)

<comment><USER9> upvoted that :)

</comment></comment></comment></comment></comment></comment><comment><USER11> Fun. I accidentally used this pattern (minus the timestamp, since it wasn't relevant for my use case) to implement a non-distributed data processing framework on top of POE. It does work pretty well, however, you need to have a way for the actor to block until enough messages have arrived (sometimes buffered operations are the most efficient). I'm not sure how you would do that with the model proposed here.

</comment><comment><USER12> This guy seems uneasy with Actors. His example is wrong and then he tries to solve it with a solution that's worse than the problem he thinks he has.

Let's untangle that stuff. Actors are about conversations between folks organized, like you would find in a good company. So let's use the conversation metaphor.

 _His example of "classical" actors:_

\- PaymentActor (to herself): Someone told me to process a payment from this order. I'm done.

\- PaymentActor: Hey, FraudCheckActor, check if this order I'm giving you is a fraud.

\- FraudCheckActor (to herself): Damn, this is a fraud.

\- FraudCheckActor: Hey, CancelOrderActor, cancel that order I'm giving you!

\- CancelOrderActor (to himself): I canceled the order.

Now I see an immediate problem here. This is anarchy. No one is in charge of the whole process, everyone runs to talk to someone else, and passing around that order data and there's no conversation happening here. Would someone in a real company organize things this way? Not in a good company, for sure!

 _Let 's see what his solution is:_

\- PaymentActor (to herself): Someone told me to process a payment for this order.

\- PaymentActor (screaming): Everyone! Payment is complete for this order! (screams the order info to everyone)

\- FraudCheckActor (to herself): I heard that, I'll check it. Damn, this is a fraud.

\- FraudCheckActor (screaming): Everyone! This order is a fraud! (screams the order info to everyone)

\- CancelOrderActor (to himself): I heard that, I'll cancel that order.

\- CancelOrderActor (to himself): I cancelled it.

Now I see an even bigger problem here. It's still a god damn anarchy, but now also _everyone is screaming at everyone_ , including the entire order data, or _talking to themselves_. How is this better again?

Plus notice, there's still state in the system - the subscription setup. The only thing achieved here is that it'll be nearly impossible to reason about how this thing works once it becomes more like a real-world scenario.

 _Here 's how you design it with Actors, once you have some experience:_

\- OrderActor: Hey, PaymentActor, process this payment and tell me what happened.

\- PaymentActor: Hey, OrderActor, I processed it. It's good!

\- OrderActor: Hey, FraudCheckActor, check this order for fraud and tell me what you found.

\- FraudCheckActor: Hey, OrderActor, damn! This is a fraud!

\- OrderActor: Hey, CancelOrderActor, cancel that order and tell me when done.

\- CancelOrderActor: Hey, OrderActor, I canceled it!

Now what do we see here? We see structure. We see clear responsibilities. We see teamwork. No screaming and no chaos. Only OrderActor has the entire order data. He's passing along only the needed bits to each actor according to their responsibilities.

And best of all, PaymentActor, FraudCheckActor and CancelOrderActor are not coupled to each other. They just respond to their events, limited to their own responsibilities.

Good job OrderActor & co.

OrderActor is a supervising actor (that's an actual term). In my example, he's managing the workflow and he is _stateful_ and as you see it's not that scary and out-of-order as the author is telling you. After all, he needs to know the payment went through, and then remember to ask for a fraud check and know what to do after that.

(Note: technically the supervisor in this example doesn't have to have state, he can just react to the events by Payment, FraudCheck, and Cancel, but this only works in this very simple example. Real world scenarios never fit into a completely stateless world. We're talking about the order _state_ here!)

The other actors have no state (that is seen in this example). So proper design isolates concerns, and may have stateless actors, but stateful actors _are_ essential to the model.

Of course, the linked article demonstrates how you can shoot yourself in the foot even with the best model out there. I find it ironic that the author was unhappy with over-complications and coupling, yet coupled everything in a complicated way with his "subscriptions" model, instead of simply using a supervisor.

<comment><USER9> There's nothing particularly wrong with his design, except for appropriating the terminology from the Actor model for what is basically CQRS.

You're just advocating for an imperative solution which may or may not be asynchronous. Nothing particularly wrong with that, but its not the only "right" way to do it. In point of fact, the underpinning of languages like Haskell is that state is conveyed by the arguments to functions (i.e. events), and the terminal "state" of a process is just the integration of those events over time or composition of functions. State is useful as a convenience. Its not essential.

Also, not for nothing, but oversimplified metaphors like shouting in an office are just obnoxious, and also irrelevant. Shouting might be a problem in an office. Software, it turns out, is capable of paying attention to whatever it wants to pay attention to and ignoring the rest.

<comment><USER12> Actually, shouting in the office means you coupled the FraudCheckActor to a PaymentCompleteEvent, and you coupled CancelOrderActor to FraudDetectedEvent.

Your lower level actors are still coupled with each other, through quite specific events, and on top of that, dependent on a subscriber setup.

Instead of declaring your own protocol and accepting messages in that protocol, now you need to understand everyone else's protocol (in the form of their event types) and react to it if you're subscribed to it.

It's the worst of both worlds.

Now, sure, you're right. Actors aren't like people in an office. Actors can't complain when you entangle them in a hot mess, they just carry on with whatever they're told to do.

But the more experience I gain, the more I see the systems of the "real world" are in fact exactly like the systems of the "computer world". If something makes no sense in an organization made of people, then almost inevitably it'll make developers cry when having to figure out the sequence of events when implementing the same organization as a set of actors.

Take it as a useful heuristic and humor my analogies. You might find a nugget of gold there.

Also, there's nothing blasphemous if my supervisor _can_ be implemented as synchronous imperative code. When people hear about a new paradigm, they believe they have to abandon everything and _do things differently_ for no reason. No. Find some balance, what you knew still works, don't throw it out, just add the new benefits Actor offers to you.

I.e. while Actors mean we can have objects that do things that you can't do in imperative/synchronous fashion, this _doesn 't_ mean we should never use actors for something that _might_ be done in imperative/synchronous fashion. 90% of the interactions between actors still look like a request/response pattern. But those 10% when things _don 't_ look like a request/response really matter when you need that ability.

Also keep in mind Actors don't care how far its subordinates are. In synchronous programming things have to happen fast, or the thread blocks. With actors a request may be sent this week, and you may get an answer next month. Things can happen very fast, or very slow, the model is not biased. How fast is fast enough is now entirely up to your app's domain logic.

Also, some Actors implement persistence (which is far more trivial than trying to persist the callback graph or call stack in "normal" code), so they don't even care if the system shuts down and it has to start over from its last state.

Imagine OrderActor sends a call to FraudCheckActor and then the system crashes, or (if it's a business workflow process) it's just powered down for the night. When the system is restarted, OrderActor is already waiting for that answer from FraudCheckActor.

Do that with imperative/synchronous code.

<comment><AUTHOR> Imperative style means that A has to know about B. In needs to know what is the next step. With reactive it does not. This creates the decoupling. And also not everyone else's protocol. An event is an important business milestones for an order which has nothing to do with the implementation, and if you look, almost all of them are as such.

<comment><USER12> Everyone has to know about the business process logic to me is the opposite of decoupling.

I'd rather limit coupling to a single actor, and let the other actors be reusable.

You can't avoid someone being in charge. It's an argument as old as the universe. Do companies need a CEO? Do we need a government?

Well, someone with an imperative role always emerges sooner or later. Even in nature, only the simplest of animals don't have a centralized nervous system. The subscription model described in the article won't survive complexity, because it's an anarchy. As nature and business shows, imperative is not a bad thing. It allows concern separation.

It's not CancelOrderActor's concern _why_ the order is canceled. The reasons for this may be hundreds. CancelOrderActor's concern is how to cancel the order. And sure, if CancelOrderActor was A.I. we could chat about why OrderActor's practice is good, but until that changes, CancelOrderActor just takes orders. It doesn't decide what's good for the order, as it's not its job.

Someone once said to me "good software design is about separating things that change a lot, from the things that don't change a lot".

Business process changes a lot. Would you rather update all actors' bindings when this happens, or just one supervisor?

Well I might be naive, but I prefer things this way: the business process changes? I change the business process actor. The payment method changes? I change the payment actor.

And not... The business process changes? Change all the things! Because it's not just a matter of whether CancelOrderActor is bound to the FraudEvent by subscription.

It's what the CancelOrderActor _does as a result of that event_ , and that's still defined _inside_ that actor. And in the real world, the options are many, because no one creates an actor for just the purpose of canceling an order anyway. It's mindless boilerplate. An actor has a domain. And how that domain changes in reaction to unrelated events depends on a lot of things.

<comment><USER9> They're equally coupled. In one system, the coupling flows outward from the business process—the business process must know about the other services. In the other, the business process is domain-driven and knows nothing about processes that are driven by the events that happen within that space. The outer processes (e.g. the fraud service) know about things that happen in the domain (or rather, what messages the domain contract will send).

I've actually written a fraud detection system this way. Worked beautifully. Within the service boundary I suppose you could say it was driven imperatively, but I don't think that's what you're talking about.

> Business process changes a lot. Would you rather update all actors' bindings > when this happens, or just one supervisor?

In point of fact, a lot of business processes never, ever change. The way they're handled may change, and at that point, I'd prefer for that concern to be nicely packaged into a service or actor or listener or whatever you want to want to model it with where my domain logic isn't concerned with it at all. And if my domain does change? I shouldn't have to change a fraud detection service at all. And in neither your model nor the OP's would I have to.

> And not... The business process changes? Change all the things! Because it's > not just a matter of whether CancelOrderActor is bound to the FraudEvent by > subscription.

I fail to see how his process has this problem and yours magically doesn't. You're making a lot of hay out of whether his "reactive" events or your "imperative" messages are better or worse, but they're the same thing: sending messages. The only difference is whether broadcast is better than a direct message sent to a unit.

<comment><USER12> > I fail to see how his process has this problem and yours magically doesn't. > You're making a lot of hay out of whether his "reactive" events or your > "imperative" messages are better or worse, but they're the same thing: > sending messages. The only difference is whether broadcast is better than a > direct message sent to a unit.

No, that's not the only difference. Notice that I have 4 actors, only one is burdened with knowing how the process works. Here's what the other 3 actors see from their PoV:

\- PaymentActor gets "process payment" event.

\- FraudCheckActor gets "check for fraud" event.

\- CancelOrderActor gets "cancel order" event.

This is what I call decoupling. 3/4 of your system is completely isolated from each other and reusable in different business processes. And the one actor who is "burdened" with knowing about the process (OrderActor) is in fact, responsible for the process (and you can open its code in one place, read it, reason about it, and change it in one place, not all over the graph).

And his model:

\- PaymentActor gets "new order" event.

\- FraudCheckActor gets "payment complete event" event.

\- CancelOrderActor gets "fraud detected" event.

Those 3 processes are coupled with each other, as they need to _interpret_ each other's events.

If you want to move to another system where the fraud check doesn't happen right after the payment, you need to edit the FraudCheckActor's "reaction" to that event.

In my case FraudCheckActor is only reacting to the "check for fraud" event and so it doesn't have to have its reaction code edited.

There's nothing "magical" about it, just better concern isolation. If you still see this as the same, that's all I could do to explain things :)

<comment><USER9> I feel like this would be a much better discussion somewhere else—i have no idea whether you're still reading this thread. :)

I get what you're saying. The way I write my systems when doing evented/reactive logic is that there is a layer between the event and the message to, say, the FraudActor, responsible for mediating between domain events from different bounded contexts (to appropriate a term from DDD) and the system in question.

The main criticism I have about your methodology is that I reject the notion that the OrderActor should be responsible for sending messages to the fraud system. An order's responsibility should be maintaining information about the order and its status—not validating whether its fraudulent, which (if the fraud detector was worth its salt) would involve a lot of historical information about the ordering party, payment information, etc—none of which should bleed into the order system, and which is the kind of thing the fraud system assembles by listening to events from all over the place. In that scenario, were the fraud system's interface to change, there would likely be changes that have to cascade to every system that touches it (regardless of whether its developed with an actor model.)

I realize you probably wouldn't, in reality and given realistic requirements, architect the system entirely the way you've outlined before this. But given the constraints of a blog post, I think the OP can be forgiven the oversimplification of the example, and you (I think) should also be able to see that his implementation has merit in real-world scenarios.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> Sorry, I couldn't get past the terrible whitish text on blackish background. Instant headache.</comment>


Mapping Arms Data – the trade in small arms and their ammunition, 1992-2011 - <AUTHOR> http://balder.prio.no/armsglobe2/index.php <comment><USER2> This was originally done by the Google Data Arts team.

info: [http://www.google.com/ideas/projects/arms- visualization/](http://www.google.com/ideas/projects/arms-visualization/) link to site: [http://workshop.chromeexperiments.com/armsglobe/](http://workshop.chromeexperiments.com/armsglobe/) github: [https://github.com/dataarts/armsglobe](https://github.com/dataarts/armsglobe)

Looks like it has been updated for 2011 data here as well as a new weapon type 'unclassified'

edit: almost forgot, here's a great writeup by Michael Chang on the project [http://mflux.tumblr.com/post/28367579774/armstradeviz](http://mflux.tumblr.com/post/28367579774/armstradeviz)

</comment><comment><USER3> What fascinates me most about these data is a single data point: Thailand.

The relationship between Thailand and Europe, and say .. Australia and Europe, over the last 2 decades, to my naive thinking, really shows details of clear product demand/response according to world incidents.

It'd be interesting to be able to overlay War/Conflict Data on top, body-count stats, and a few other relevant data of the world, to see how all this is tied to a market reality.

Perhaps we, the people, need to do our own data-mining.

</comment><comment><AUTHOR> Here is the BBC coverage of the tool: [http://www.bbc.co.uk/news/magazine-23822086](http://www.bbc.co.uk/news/magazine-23822086)

</comment><comment><USER4> Interesting to note that during the Obama years the US has dramatically increased it's import to export ratio.

<comment><USER5> I think that's as much a function of winding down the war in Iraq as anything. Great eye candy, but I would like to have had some more histograms to present the historical context more clearly.

<comment><USER6> It's due to the "He's gonna take my guns" attitude that has permeated the fanatical right wing of the US population. Gun and ammo sales has been a growth industry since 2009. Of course they're just laughing the whole way to the bank since no one is going to take their guns.

<comment><USER7> You deny "He tried to take our guns" after his last (re)election, when he had more "flexibility"? That major political figures like Dianne Feinstein (who also cares nothing about our privacy; strange, that) and Andrew Cuomo didn't call for gun confiscation? That ones like Cuomo and the Democrats in Colorado didn't get get new very nasty gun "control" bills passed (e.g. you can now only load 7 rounds in your magazine when using it for self-defense), where gun owners who bought various stuff prior to them were grandfathered?

The utter ruthlessness of Fast and Furious is also telling.

You're also ignoring what I think is the most fundamental thing: anyone who makes a habit of balancing their checkbook can predict that things are going to get ugly when we can no long have "trillion dollar deficits as far as the eye can see", and that we're clearly going to be on our own as that plays out?

How about HUD's Section 8 policy that among other things moves criminals out of the central cities into formerly peaceful areas, which is now slated for a big expansion?

Another thing, related to the above, a steadily graying population, etc., is that ever more states have shall issue concealed carry regimes. Since Obama was elected Iowa and Wisconsin have joined them, with Illinois (!) in a few months; that'll make 43 out of 50 states, way more than 2/3rds the population.

You're demonstrably wrong about "no one's going to take our guns" when it comes to slave states like New York and California that have confiscated them in recent memory, and in the latter's case is poised to do another round, but in general correct ... because in most of the US we won't tolerate it, one way or another. Keep pushing, though, if you want a second Civil War with the half or so of the nation that's "the fanatical right wing".

</comment></comment></comment></comment><comment><USER8> Almost every country is covered, very interesting data, You can follow up who is connected to what. And timetable provides very interesting picture how countries under imminent treat imports arms.

</comment><comment><USER9> The json it loads is 4MB and takes about 2 minutes for me. The graph is kind of hard to make out with everything selected.

<comment><USER10> I also find the graphics to be a little bit difficult to read.

It is hard to tell where the lines lead, so I need to rotate the globe. Also for European nations, it is very messy because some of the countries are tiny and all trade with each other. There is also no way to see arms trade over time, which would have been quite useful since the data has a time-series.

Very pretty looking 3D graphic though.

</comment></comment><comment><AUTHOR> Looks like it's done using D3.

<comment><USER2> Just for a few of the graphs, majority of the 'wow factor' is webgl/three.js</comment></comment>


How to Model Viral Growth: Retention and Virality Curves - <AUTHOR> http://www.linkedin.com/today/post/article/20130402154324-18876785-how-to-model-viral-growth-retention-virality-curves

<comment><USER2> Definitely an interesting article and one worth reading. I work a lot on viral growth and I'd like to add a counterpoint that the growth of apps is so complicated that it is basically impossible to model in a useful way.

I've found that even if I make no changes to an app, the retention and virals fluctuate quite a bit for no apparent reason, and the fluctuations are big enough that it makes long term forecasting really more of guesswork than anything else.

Also, there are second order effects that are hard to model as well. For instance, improving virals can improve retention (user A invites friend B, user A stays for longer because their friend uses it).

I've gone through the process of modeling a couple apps, and it quickly gets to a point where the relationships become circular and small variations cause exponential differences down the line.

It is important to make informed decisions about virals and retention, but I don't think such a model is the way to do it. I think it is more important to think about optionality and decision making in opaque environments rather than trying to model the unmodelable.

</comment><comment><USER3> This multiplicative kind of model is happily amenable to time series analysis, so you can do stats to see what your numbers are and how well they fit. That's great. What's less great is the model quality, given that well-tested virality models can be found in other venues. Coffman looks at this, for instance, at [http://datacommunitydc.org/blog/2013/01/better-science-of- vi...](http://datacommunitydc.org/blog/2013/01/better-science-of-viral- marketing/). The difference in these two types of models is that symmetries in the statement of the problem permit, or exclude, classes of solutions. Those symmetries come from assumptions about the contact graph, the most basic (and testable) assumption.

</comment><comment><USER4> This is great stuff. I see so many startups get so excited about features and growth yet fail in their analysis of retention. Maybe I'm a bit on the data- nerd side but I love to see folks sharing their own methods for tracking and calculating this stuff. Even with so many startups basing their model on recurring revenue, it's still easy to trip up on modeling this stuff going forward.

<comment><USER5> > so many startups get so excited about features and growth yet fail in their > analysis of retention

I think it's just we write less about it, not so much that we aren't aware of its importance. "How we managed to retain our users for 4 months" sounds admittedly less sexy than "How we got a bazillion users in less than 72 hours", but the truth is a tech startup with no strong retention strategy is basically dead in the water, and generally folks know this.

</comment></comment><comment><USER6> Here's a different approach:

We denote time by t with units, say, days.

The number of customers at time t is the (real valued function of a real variable) y(t).

We assume that at the present t = 0 and that we have y(0), that is, the current number of customers.

We let the number of customers who will ever try our business be b. That is, b is our intended 'market potential'.

Initially we assume that once we get a person as a customer, we do not ever lose them but keep them forever.

As usual, we let y'(t) = dy(t)/dt be the calculus first derivative of y(t). Then y'(t) is number of new customers per day, that is, the 'rate' at which we gain customers.

For 'virality' we notice that that is proportional to (1) the number of customers y(t) we have 'talking' about our business and (2) the number of people

         b - y(t)

yet to be our our customers hearing the talking.

Then we have that for some constant of proportionality k

         y'(t) = k y(t) (b - y(t))

So we have an initial value problem (that is, we know y(0)) for a first order (we use only the first derivative) ordinary (no partial derivatives) differential equation.

Then from calculus,

         y(t) = y(0) b exp(bkt) /
                ( y(0)( exp(bkt) - 1) + b))

So this solution grows (1) initially slowly, (2) then more rapidly, (3) then more slowly and approaches b asymptotically from below.

In case we lose some customers forever at some rate r, then we get the same solution except k and b get adjusted.

Once there was a startup (now a major company) that was struggling and had as an investor a major company with a Board seat and at the startup two representatives, one in finance and the other in aeronautical engineering.

The two representatives had asked for some revenue growth projections.

People around the HQ considered what the startup hoped, intended, thought might happen, etc., but found nothing credible.

One guy who remembered calculus reluctantly got involved, formulated and solved the differential equation above, and showed the solution to a Senior VP of Planning (SVP) who reported to the founder, CEO, COB. The SVP was responsible for the projections. The SVP took the guy's calculus solution as the basis of the projections and on a Friday sat with the guy with a pocket calculator and some graph paper and graphed solutions to the differential equation for selected values of the constant k and picked one of the solutions as the official projection.

The next day, Saturday, at about noon, the guy was in his office working on some other math problems and got a call from a person asking if he knew about the projections for the Board and if he could come over to the HQ? Sure. When the guy arrived, the situation was grim: The two representatives of the major Board Member were standing in the hall with their bags packed with airline tickets back to Texas. The startup was about to die.

The SVP was traveling and out of town.

The person who had called got the graph of projections from the previous day and asked the guy to reproduce a point on the graph. Using the calculator, the solution above, and a few keystrokes, the point on the graph was reproduced. After several more points were reproduced, the area became happier; the two representatives on the Board stayed, and the startup was saved.

Later the person who had called explained that that Saturday was a Board meeting, the growth projection graph was shown, and the two representatives had asked how the projections were calculated. The rest of the company tried to reproduce the graph but could not. The Board meeting stopped. The two representatives lost patience with the startup, got airline tickets back to Texas, returned to their rented rooms, packed their bags, and as a last chance returned to the startup to see if there was an answer to how the projections were calculated.

Ah, one saved startup! One reason to take calculus seriously!

<comment><USER6> Note that with this derivation, if accept the assumptions (which obviously do not always hold), then all there is to 'viral' growth are three numbers, the current number of customers y(0), the eventual number of customers b, and the constant k. This situation holds also in the case of some customers leaving and never coming back (just by some adjustments in b and k).

For k, might fit to past data. For given y(0) and b, all k does is adjust how fast the curve rises to the asymptote. So basically all we are doing is interpolating between y(0) and b.

Otherwise, all viral curves are the same.

So, an advantage of my derivation is a simple, explicit equation for a fairly general solution.

The article has a comment claiming that biology addresses a similar problem and gets a 'logistic' curve. The comment didn't say just what was meant by a logistic curve, but I suspect that my solution here is an example. If so, then here we have an 'axiomatic' derivation of the logistic curve.

It is true that the growth of some products, e.g., TV sets, look to the eye very much like one of the curves from my solution for selected values of y(0), b, and k.

Could also make a Markov assumption: So, assume that get new customers (and, if wish, lose old customers) at some 'rates' and, thus, get a continuous time, discrete state space Markov process. Then as is well known the solution is a matrix exponential. Could evaluate the matrix exponential or just use Monte Carlo to generate a few thousand sample paths. Then could put some confidence limits on the deterministic solution.

Since no one guessed the war story, the startup was FedEx, the SVP was Mike Basch, the CEO, of course, was Fred Smith, the person who called on the phone was Roger Frock, and the investor was General Dynamics. The arithmetic was courtesy of an HP-35. So, HP might run an ad saying how they saved FedEx!

</comment></comment><comment><USER7> As others have pointed out, this can modelled with calc pretty handily.

I found agent-based modelling much more interesting. For example, mean-field models struggle with non-uniform spaces.

For the interest of persons here attending, I've uploaded my crappy code on this topic. It's 3 years old and not production suitable.

<https://github.com/jchester/ruby-epidemic-model></comment>