Why Did It Take Devuan Two Years to Replace Systemd? - <AUTHOR> http://troubleshooters.com/linux/systemd/bikebrakes.htm <comment><USER2> I keep waiting for the reasoned, technical arguments against systemd instead of veiled metaphors, complaints about its creator, or protestations that it's different.

The initd-inspired community has resisted the push for declarative and component-based core system infrastructure for so long, and it's causing very strange distortions to the linux ecosystem as solutions get sought for the problems it introduces.

Systemd, for all its flaws, fixes some very important problems that init and runit and similar systems left unsolved. It's complicated, but it's hard to argue it's unnecessarily so. We always end up at these tortured bike metaphors or some other garbage. There's a whole website devoted to criticisms and it feels like at least half of the site is devoted to this kind of baseless rhetoric.

As far as I can tell, systemd is a holy war. It's the new editor debate. It's vi vs emacs vs sublime vs atom vs ... down throughout time, with people arguing where the tradeoff between simplicity of execution is more important than delivering a system people asked for.

<comment><USER3> One of the main problems was the speed with which systemd was rolled out on all the major Linux distros, despite it being relatively immature and very controversial at that.

The second major issue is it growing way out of the bounds of being just an init replacement, thereby violating the so-called Unix Philosophy of doing one-thing well. For a user app to do that is one thing, but for a core OS component to do that just smells way too much like forcing Linux to be more like Windows, which a lot of hard-core Linux users are ideologically opposed to.

The third major issue is that on most major Linux distros, users are not given an easy way to avoid using systemd, no matter how much they hate it or oppose it.

The fourth issue is the apparent arrogance of the people responsible for systemd, and their off-hand dismissal of mature and widely respected unix conventions.

All of this adds up to a storm of controversy that the systemd people mostly brought upon themselves. Had they just been more humble about their creation and waited until it was mature and well-tested and did all the wonderful things they claim it could and should do instead of stuffing what was widely seen as a broken-by-design piece of garbage down everybody's throat, maybe much of the Linux community wouldn't have been nearly so outraged by it.

<comment><USER4> _The second major issue is it growing way out of the bounds of being just an init replacement, thereby violating the so-called Unix Philosophy of doing one-thing well._

I keep on hearing this overly worn "Unix Philosophy" argument against systemd, which in the absence of any non-religious technical justification, basically boils down to "its different".

 _All of this adds up to a storm of controversy that the systemd people mostly brought upon themselves. Had they just been more humble about their creation and waited until it was mature and well-tested and did all the wonderful things they claim it could and should do instead of stuffing what was widely seen as a broken-by-design piece of garbage down everybody 's throat, maybe much of the Linux community wouldn't have been nearly so outraged by it._

Nobody shoved anything down anyone's throat, least not the SystemD developers. And the same goes for PulseAudio, DBUS, and all the other projects on freedesktop.org.

Anyway, in reality, software does not become mature and well-tested without early exposure to real users and systems. Chicken-and-egg problem there.

I feel absolutely no sympathy for those who are outraged. There are many distributions out there (including Devuan) which align with their particular vision of an OS should be.

<comment><USER5> Why does an init system need a DNS implementation? Why not use named, unbound, or any of the other implementations?

Instead they implement it poorly, have had various problems with it, and of course there have been security problems with it.

Do you really want an init system that replaces a ton of functionality poorly like dns, ntp, syslog, xinetd, etc etc etc?

<comment><USER4> > _Do you really want an init system that replaces a ton of functionality > poorly like dns, ntp, syslog, xinetd, etc etc etc?_

SystemD is not just an "init" system. You have to look at the larger vision for it being a set of building blocks for Linux system and service management.

I think this is a very good thing. Its about time that Linux had a unified and self-consistent approach to system management and configuration.

<comment><USER3> _" SystemD is not just an "init" system. You have to look at the larger vision for it being a set of building blocks for Linux system and service management."_

This was not the way it was sold. It was originally sold as an init replacement.

If at the start the systemd guys had come out and said, "We aim to replace much of the core of Linux with some kind of Windows-Linux conglomoration: replace not only init but dns, syslog, turn all system log files binary, replace intetd, fuck with account naming, and oh, by the way, there'll be no easy way for distros that go with systemd to give their users a choice to use something else" then I'm not so sure how many people would have gotten onboard.

Basically, systemd has been a retooling of the core Linux ecosystem by people who think they know better. Well, this might be news for systemd fans, but they don't necessarily know any better than the rest of us, and the rest of us would like some choice in the matter. Instead we're force fed systemd and our operating system are no longer Linux but some bastardization of Linux and Windows.

 _" Nobody shoved anything down anyone's throat, least not the SystemD developers."_

Tell that to the significan number of RedHat, Debian, and CentOS users who strongly object to systemd yet are forced to use it because that's all their distro supports.

 _" Anyway, in reality, software does not become mature and well-tested without early exposure to real users and systems. Chicken-and-egg problem there."_

You don't have to test it out on all RedHat users, all Debian users, all CentOS, etc, etc. Getting a bunch of _volunteers_ and letting them use a feature-complete version for five or six years before springing it on the rest of us would have been a good start. Instead you get an ever-morphing and ever- growing trojan horse.

<comment><USER6> You will likely have this argument again so I need to point out two things:

1) Debian does support sysvinit right now, although with some packages having hard dependencies on systemd it might not be long before you hit a brick wall.

2) systemd (the init) was used by Fedora for a long time before getting into RHEL/CentOS- which is of course before it got into Debian. So there were volunteers.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> Sure, it might be because System D integrates so tightly with the whole ecosystem that it is hard to replace it.

Or: Those folks who write rants about system d are not very active in contributing to a distribution that runs without it. With some discussions, you have that feeling that there are "those who talk, and those who do".

<comment><USER8> Or maybe we just gave up on Debian, and decided to spend our efforts somewhere else?

I've been fairly vocal against systemd, and not because of Poettering. I'm no fan of the man, and I strongly agree with a lot of the points he made in his initial blog posts when introducing systemd. I think he's probably very talented---I'm not in a position to judge that. I wouldn't be against systemd if it were _just_ an init system. The problem is that it's not. That's the end of my argument. If systemd were just an init system, I'd be very happy to have it on my machines. Poettering even admitted this himself [1]:

> Well, systemd certainly covers more ground that it used to. It's not just an > init system anymore, but the basic userspace building block to build an OS > from, but we carefully make sure to keep most of the features optional. You > can turn a lot off at compile time, and even more at runtime. Thus you can > choose freely how much feature creeping you want.

In response, I've decided to not use systemd on my machines. I'm a patreon of xtraeme [2], the creator of Void Linux [3] and xbps (the package manager used in Void). Void Linux uses runit [4], which is exactly as simple and fast as I'd like it to be [5]. Working in startups, I haven't had time to contribute code to either xbps or Void, but at least I'm contributing in the way that I can. I didn't contribute to Devuan (other than seeding a few hundred TB of ISOs) for the same reason that I haven't contributed to Debian in a long time: I don't use it any more.

[1]: [http://0pointer.de/blog/projects/the-biggest- myths.html](http://0pointer.de/blog/projects/the-biggest-myths.html)

[2]: [https://www.patreon.com/xtraeme](https://www.patreon.com/xtraeme)

[3]: [https://www.voidlinux.eu/](https://www.voidlinux.eu/)

[4]: [http://smarden.org/runit/](http://smarden.org/runit/)

[5]: [https://www.voidlinux.eu/usage/runit/](https://www.voidlinux.eu/usage/runit/)

<comment><USER9> > I wouldn't be against systemd if it were just an init system. The problem is > that it's not. That's the end of my argument.

All of systemd's functionality, as far as I'm concerned, is rationally motivated by wanting to orchestrate and instrument system startup in a sensible way. What functionality do you think systemd should not have that it has today? What is a better way of achieving the result provided by that functionality?

If your perspective is that you don't care about that functionality or the related user experience, and so all of its complexity is dead weight to you, OK. That makes sense. But as a software engineer and system administrator, I find the features extremely useful, and I think many other people do too. That's why systemd has been adopted by major distributions.

I wrote a comment earlier [1] about systemd that reviews Russ Allbery's analysis of systemd [2], from when Debian was evaluating switching to systemd and upstart. Russ's comment reviews a number of different systemd functions, such as its integrated journal, and then explains the pros and cons of each. Russ even said:

        Integrated daemon status.  This one caught me by surprise, since the
        systemd journal was functionality that I expected to dislike.  But I was
        surprised at how well-implemented it is, and systemctl status blew me
        away. [lots more details in linked comment]

[1] [https://news.ycombinator.com/item?id=13387989](https://news.ycombinator.com/item?id=13387989) [2] [https://lists.debian.org/debian- ctte/2013/12/msg00234.html](https://lists.debian.org/debian- ctte/2013/12/msg00234.html)

Sure, you could argue an init system "shouldn't" have an integrated journal (for example), but then you'd lose the clear benefits that Russ outlines. You wouldn't achieve the same benefits or user experience. Similarly, systemd's configuration-driven approach to service definition makes it possible to employ a wide variety of standard configuration options across all services. From my previous comment:

> I can launch my service at the appropriate time during boot with > configuration as simple as:

      [Unit]
      Description=Demo service

      [Service]
      Type=forking
      ExecStart=/usr/sbin/my-daemon

> Now let's say that I didn't author this daemon, but I'd like to run it with > a private network, private temp folder, or a private /dev namespace. Or > perhaps the daemon needs to run as root, but I want to drop all capabilities > it doesn't need. It's as simple as adding these lines to the service's > configuration:

      PrivateTmp=yes
      PrivateDevices=yes
      PrivateNetwork=yes
      CapabilityBoundingSet=CAP_NET_BIND_SERVICE

> The fact that systemd supports these configuration options means that > there's a simple and standard way to employ them with any service. The > service itself doesn't need to support them, and needn't complicate its own > daemonization logic to do so correctly. Indeed, I don't need to trust the > service to daemonize or drop capabilities, since I can tell the init system > do that before launching the service.

> I can drop capabilities with CapabilityBoundingSet=, or limit resource usage > with CPUSchedulingPriority=, IOSchedulingPriority=, etc. I could even tell > systemd to open the listening socket for me so the service doesn't need > CAP_NET_BIND_SERVICE! Moving these options into the init system makes a ton > of sense, because it gives administrators the ability to employ these > features from outside applications, not just by enabling them within > applications that bother to explicitly support them via command line > arguments. Systemd better encourages the principle of least privilege: if a > system daemon does not need the ability to "ptrace" other processes, or bind > to ports <1024, then as the administrator I can take those away with > CapabilityBoundingSet= in the unit file. Chrooting the service is as easy as > RootDirectory=. This is a huge step forward compared to the world where > every service must be relied upon to expose these settings, and must be > trusted to implement them correctly.

<comment><USER10> > All of systemd's functionality, as far as I'm concerned, is rationally > motivated by wanting to orchestrate and instrument system startup in a > sensible way. What functionality do you think systemd should not have that > it has today?

QR code generator? Message broker (IPC)? Underdeveloped, brittle, and overcomplicated syslog replacement? Task scheduler?

<comment><USER9> > Underdeveloped, brittle, and overcomplicated syslog replacement?

Are you referring to the systemd journal? I think you should lay out your arguments in detail if you want to make claims like that, since there is a lot of compelling evidence to the contrary. Russ Allbery's analysis of systemd [1] lays out a convincing case that the journal is a highly developed, flexible, and useful feature:

      * Integrated daemon status.  This one caught me by surprise, since the
      systemd journal was functionality that I expected to dislike.  But I was
      surprised at how well-implemented it is, and systemctl status blew me
      away.  I think any systems administrator who has tried to debug a
      running service will be immediately struck by the differences between
      upstart:

        lbcd start/running, process 32294

      and systemd:

        lbcd.service - responder for load balancing
         Loaded: loaded (/lib/systemd/system/lbcd.service; enabled)
         Active: active (running) since Sun 2013-12-29 13:01:24 PST; 1h 11min ago
           Docs: man:lbcd(8)
                 http://www.eyrie.org/~eagle/software/lbcd/
       Main PID: 25290 (lbcd)
         CGroup: name=systemd:/system/lbcd.service
                 └─25290 /usr/sbin/lbcd -f -l

      Dec 29 13:01:24 wanderer systemd[1]: Starting responder for load balancing...
      Dec 29 13:01:24 wanderer systemd[1]: Started responder for load balancing.
      Dec 29 13:01:24 wanderer lbcd[25290]: ready to accept requests
      Dec 29 13:01:43 wanderer lbcd[25290]: request from ::1 (version 3)

      Both are clearly superior to sysvinit, which bails on the problem
      entirely and forces reimplementation in every init script, but the
      systemd approach takes this to another level.  [...] the most useful
      addition in systemd is the log summary.  And that relies on the
      journal, which is a fundamental  design decision of systemd.

      And yes, all of those log messages are also in the syslog files where
      one would expect to find them.  And systemd can also capture standard
      output and standard error from daemons and drop that in the journal and
      from there into syslog, which makes it much easier to uncover daemon
      startup problems that resulted in complaints to standard error instead
      of syslog.  This cannot even be easily replaced with something that
      might parse the syslog files, even given output forwarding to syslog
      (something upstart currently doesn't have), since the journal will
      continue to work properly even if all syslog messages are forwarded off
      the host, stored in some other format, or stored in some other file.
      systemd is agnostic to the underlying syslog implementation.

[1] [https://lists.debian.org/debian- ctte/2013/12/msg00234.html](https://lists.debian.org/debian- ctte/2013/12/msg00234.html)

The benefits that Russ describes above are meaningful, tangible improvements to the user experience that helps system administrators like me. Far from being brittle, the journal seems quite flexible (being "agnostic to the underlying syslog implementation"); it seems highly-developed and well- integrated into the systemd toolkit, and I see no evidence that it's over- complicated.

> Message broker (IPC)?

Are you referring to dbus? Dbus is not part of systemd and predates systemd by almost a decade! Dbus already had significant adoption when systemd took a dependency on it. It's used by GNOME, KDE, and even the Upstart init system. From Lennart Poettering's blog [2]:

      D-Bus has been a core component of Linux systems since more than
      10 years. It is certainly the most widely established high-level
      local IPC system on Linux. Since systemd's inception it has been
       the IPC system it exposes its interfaces on. And even before
      systemd, it was the IPC system Upstart used to expose its
       interfaces. It is used by GNOME, by KDE and by a variety
      of system components.

[2] [http://0pointer.net/blog/the-new-sd-bus-api-of- systemd.html](http://0pointer.net/blog/the-new-sd-bus-api-of-systemd.html)

<comment><USER10> > [...] Russ Allbery's analysis of systemd [...] explains why it's such a > compelling feature:

It's tightly integrated where it doesn't need to be (with zombie reaper), you can't replace it (only slap something on top of it), and generally those few last log lines that are displayed are useless anyway, as usually when something breaks I still need to view full logs. Much more useful would be debuggability (with syslog this part is easy), remote data collection (syslog has it for decades already), and storage resilience (journald can corrupt log files on system crash).

> Are you referring to dbus? Dbus development began in 2002 whereas systemd's > first release was in 2010.

When what was started is pretty much irrelevant. I didn't say we don't need D-Bus as a message broker (though I consider sub-par the way D-Bus works), I said that we don't need it in neither zombie reaper nor daemon manager. It worked well enough being a stand-alone service for a decade.

And now, you haven't addressed the glorified zombie reaper including a task scheduler and QR code generator. Why do you think their integration is so important that they needed to be included in systemd?

<comment><USER9> Any discussion about this should be anchored with an answer to the questions I asked earlier: What functionality do you think systemd should _not_ have that it has today? What is a _better_ way of achieving the benefits provided by that functionality?

Let's talk about logging. The systemd journal is designed to capture and log the output of all programs. Some programs do not integrate with syslog: they write to stdout/stderr instead. As a system administrator, I desire a way to capture the logs of everything I run on a system, including programs that do not integrate with syslog (e.g., ad hoc shell scripts and applications).

Who is in the best position to capture program stdout/stderr? The process supervisor. Systemd does that with its journal facility, and other supervisors work similarly. Personally, I value having a uniform method for capturing program logs from all processes on a machine, and that's what systemd offers me. Syslog by itself does not achieve this goal.

How do you recommend capturing and logging program output, if not by having the supervisor do it? (And why is that approach preferable?)

The journal is an integrated part of the systemd toolkit, and that's a good thing because it enables useful capabilities like capturing stdout and good user experiences like `systemctl status`. But you don't have to interact with it. If you prefer the syslog toolchain instead, then redirect the journal to syslog via ForwardToSyslog=yes in journald.conf. The journal still adds value with this approach because it captures logs that syslog can't (and without kludges like `logrotate` and `logger`).

You can see more log lines with e.g. `systemctl status -n50`. If you want to see everything, invoke `journalctl -u service-name`.

I view the systemd journal as a logging facade, a _complement_ to facilities like syslog, not an _alternative_ to it. As Russ said in my quote above, "systemd is agnostic to the underlying syslog implementation."

I'm not sure I grasp your point about dbus. I have not seen anyone make the argument that dbus should be part of a zombie reaper or process supervisor. However, IPC is a useful and sensible way for processes on the system to communicate with their process supervisor, which is why both systemd and upstart integrate with dbus. I've read that it is possible to use systemd without dbus [1] (there are apparently Debian configurations that work this way), though I haven't deeply looked into it.

If you want to launch supervised processes with clean environments, namespaces, and cgroups, then it is necessary for a system process to be the one that forks. Communicating that request between processes necessitates IPC. As the most widely adopted IPC system for Linux, dbus was an appropriate choice for systemd and upstart to integrate with. Again, going back to our anchors: Is there another IPC framework you think systemd should have integrated with instead? Or if not by using IPC, how do you think service management commands (e.g., restart a service) should be communicated to the system?

[1] E.g. [https://github.com/mitchellh/vagrant/issues/7973](https://github.com/mitchellh/vagrant/issues/7973) [https://github.com/aelsabbahy/goss/issues/101](https://github.com/aelsabbahy/goss/issues/101)

<comment><USER10> > Any discussion about this should be anchored with an answer to the questions > I asked earlier: What functionality do you think systemd should not have > that it has today? What is a better way of achieving the benefits provided > by that functionality?

I already replied to that.

> How do you recommend capturing and logging program output, if not by having > the supervisor do it? (And why is that approach preferable?)

Supervisor is in appropriate place only for substituting STDOUT and STDERR descriptors at the daemon's start, but once that is done, the descriptors are easily transferrable to logging daemon. Obviously, the logging daemon should understand that, so rsyslog would need a plugin to accept those descriptors, or a generic intermediate daemon that reads all the output and sends it further to syslog would be needed, but that's it. There's literally no benefit in coupling log collector with zombie reaper as tightly as in systemd.

> The journal is an integrated part of the systemd toolkit, and that's a good > thing because it enables useful capabilities like capturing stdout

...which is easily achievable without coupling...

> and good user experiences like `systemctl status`.

...which is useless anyway, as I said in my earlier post. Oh, but this "good user experience" is indeed really excellent, because when I start a daemon and it dies, `systemctl start' does not print the information that daemon failed to start nor the daemon's initial logs. I get _nothing_ and I _don 't know_ that the daemon died. Indeed, great user experience. Or updating unit files, again, great experience when the unit file acts in exactly the same way as previously, because I forgot to reload systemd and it didn't inform me to do so. Or systemd automatically chopping log output to my terminal. Or awful command line syntax of journalctl. Top notch user experience indeed!

> But you don't have to interact with it. If you prefer the syslog toolchain > instead, then redirect the journal to syslog via ForwardToSyslog=yes in > journald.conf.

So this tight coupling is good, because it gives me nothing useful over not coupled architecture and I can easily work around this coupling? This is what you're trying to say?

> The journal still adds value with this approach because it captures logs > that syslog can't (and without kludges like `logrotate` and `logger`).

First, syslog _can_ capture this output. It's not impossible and it's not even difficult.

Second, these "kludges" that _worked well for decades_ are replaced with (a) brittle disk format that breaks apart on system crash, (b) lack of control over how long the logs are kept, and (c) lack of remote log collection.

> Is there another IPC framework you think systemd should have integrated with > instead?

You're asking wrong question. It's not that systemd should not _integrate_ with D-Bus, it's that it should not _swallow_ it.

And now I'll ask _once again_ : what is the so important reason for systemd to provide QR code generator and task scheduler?

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> This is why I've not been so enthusiastic about the systemd project. When I first heard of systemd, it seemed like a good idea: A new clean-slate design of an init system and service management that aimed to address shortcomings of the present common init and service management. And had the systemd project stayed just that, I'd have been fine with it, and may well have welcomed it onto my systems with open arms.

But then it became necessary to subsume udev and syslog and cron and more parts of the system, and each time, the justifications for this scope creep have been unconvincing to me. It comes across as a bunch of added complexity because it'd be nifty and because they can, without stopping to reflect on whether trying to subsume the rest of the OS or forcing tight coupling and integration is a good idea from a design or architecture perspective.

</comment><comment><USER12> Say I want to create an independent, 100% compatible implementation of systemd which can just drop-in to any systemd installation, no matter how it is configured and have it work?

Where can I download the precise set of requirements for this, so that I don't have to peek at the systemd source code?

 _What_ is systemd?

<comment><USER13> Say I want to create an independent, 100% compatible implementation of Linux which can just drop-in to any Linux installation, no matter how it is configured and have it work?

Where can I download the precise set of requirements for this, so that I don't have to peek at the Linux source code?

What is Linux?

Really, this doesn't make sense.

<comment><USER14> How can people drop a comparison between a kernel and a supposed init and not stop and go "wait, what?!".

Having a would be second kernel metastasizing in userspace is no a good thing, full stop.

</comment></comment></comment><comment><USER15> The linked page does not actually answer the question. Instead we get a long bicycle analogy that does not really map to systemd in any way that I can tell.

It would have been much more interesting to hear about what software had to be rewritten or replaced to support desktop environments like Gnome.

</comment><comment><USER16> I mean, it is kind of strange that with one apt-get install, one apt-get purge, one apt-get autoremove, and one pinfile in /etc/apt, you can replace systemd in Debian. You'd think Devuan could just do that and redistribute or something.

<comment><USER17> It's not systemd that's the problem, it's libsystemd0. Try removing it on Stretch. It's doable [0] but it's a pain and makes future maintenance fragile and error prone. And if I understand correctly once you do so you're not running Gnome on that system anymore...

[0] [http://lkcl.net/reports/removing_systemd_from_debian/](http://lkcl.net/reports/removing_systemd_from_debian/)

<comment><USER18> Which is a purely religious problem as libsystemd0 basically does nothing if systemd is not running. Debian is a binary distribution, a lot of stuff is linked against libs you probably never use, that's just how it is.

<comment><USER17> I didn't say it was being used but the Devuan project wants it out of their distro and dependencies on it are much harder to resolve than dependencies on systemd bins.</comment></comment></comment></comment>


Ask HN: Career path guidance - <AUTHOR> Background:
I am in my second to last year of my undergraduate degree studying computer science and engineering. I have been told that the next best step is to get a technical graduate degree. My guess, however, is that by getting a more technical degree, I will be locked into a more technical position. Front-end development (UI/UX), marketing, and management interest me greatly. As I look at career options, I am realizing how much more of an advantage you have if you know which career path you want to take early on, so you can tailor your experience/skills.
Goals:
The career I consider ideal would be working in a position where: 1. I have more influence, which I suppose would be at a start-up or a small/medium enterprise 2. Am able to work with multiple factions within a company (be involved with the whole process) 3. Am able to grow professionally
Questions (many of my friends have the same unanswered questions with a general regard to the tech industry): What would you recommend for someone with my background in terms of career paths?
Does a technical graduate degree make that big of a difference?
Would an MBA (possibly in marketing) be beneficial?
Any advice would be greatly appreciated. <comment><USER2> I have a neighbor who is a full time artist. Her income sources are diverse. She teaches workshops. She provides private lessons. Sure she paints paintings and sells them, but those paintings are painted with content and technique that sells not for radical experimentation in artistic expression: some are even done pseudonymously, scanned and giclee'd "in bulk" for interior designers...and the nice thing about those is that checks come in the mail long after the work is done.

The reality of a particular career is usually unrelated to the way a person thinks about it in school. Beyond technique, there's nothing much in her career that was taught in school. The strong relationship between her work and her study is her passion for painting.

If you're doing MBA type stuff or UI/UX work in your free time, those are great ways to extend your interest. If you're currently managing and running organizations, then that's a sensible basis for pursuing more knowledge and training. It's also reasonable to realize that you don't like doing technical work: many young people are pushed toward it in lieu of their deeper interests.

My advice is to think about what you _really_ want to do. I doubt it is "being an MBA".

Good luck.

<comment><AUTHOR> Thank you very much for your response. It is insightful and though provoking and I appreciate you taking the time to write it.

</comment></comment><comment><USER3> I have no specific advice for you, but I think you are wrong thinking that having skills in different areas will hinder you.

On the contrary, remarkable people I have been working with, are usually good in more than one area. Being a good product manager is nice, being a good product manager, and having the depth of knowledge necessary to understand intimately how the product works and how it could be improved is awesome.

<comment><AUTHOR> I agree with what you are saying. I probably should not get tunnel vision towards a particular career and appreciate what it means to be well-rounded. Thank you for your response.</comment></comment>


Artificial Intelligence Needs a Bullshit Meter - <AUTHOR> https://gab41.lab41.org/i-need-an-ai-bs-meter-27e94d48c8c1#.omnk0unal <comment><USER2> This is a very confused article with many errors. It appears to imply that no one validates the accuracy of machine learning models, or that they lie about the accuracy. Even more strangely, it suggest that the only way to address this is by ensemble methods.

Validation of models is one of the most important parts of any machine learning system. Every expert practitioner measures the accuracy of their models with well established methods such as cross validation or hold out tests. So the basic premise of this article seems quite at odds with the reality.

Further, the article users the security domain as an example of the lack of validation. Most applications of ML to security use _unsupervised_ algorithms to perform anomaly detection. This is entirely a different thing to a supervised algorithm. Anomaly detection via unsupervised algorithms is well known to have many false positives.

But possible the worst error in the article is suggesting that ensemble methods are a way to validate the accuracy of a model. An ensemble technique is _not_ a way to validate accuracy. Rather it's a way to try to obtain higher accuracy. You still need to validate your ensemble via something like cross fold validation to understand the expected error.

<comment><USER3> _This is a very confused article with many errors._

Au contraire! It is a good article which highlights a number of subtle points!

 _But possible the worst error in the article is suggesting that ensemble methods are a way to validate the accuracy of a model. An ensemble technique is not a way to validate accuracy. Rather it 's a way to try to obtain higher accuracy._

Err... to quote the article: "One of the ways to _improve result quality_ is by running ensembles of algorithms."

The talk of blending recommender systems and deep learning appears to be inspired by Google's Wide and Deep Learning[1] work, which is effectively a way of blending global and local results.

 _Every expert practitioner measures the accuracy of their models with well established methods such as cross validation or hold out tests._

The problem here is knowing how well the model will work with radically (or even somewhat) different data than it was trained on. This is not the same as doing CV or hold out.

For example, the ImageNet set has an enormous number of dog pictures. This means that its CV or hold-out performance tends to translate well to performance on similar datasets, and if the new dataset has a lot of dog pictures it will translate very very well.

However, if you attempt to use a network trained on ImageNet in a completely different context (classifying X-Rays for example) it is unclear how well it will perform before testing.

 _Further, the article users the security domain as an example of the lack of validation. Most applications of ML to security use unsupervised algorithms to perform anomaly detection. This is entirely a different thing to a supervised algorithm. Anomaly detection via unsupervised algorithms is well known to have many false positives._

Lab41 works in the intelligence space. That isn't your normal computer- security anomaly detection. Have a look at their other work[2] - there is only one thing that is conventional security log file analysis.

[1] [https://research.googleblog.com/2016/06/wide-deep- learning-b...](https://research.googleblog.com/2016/06/wide-deep-learning- better-together-with.html)

[2] [http://www.lab41.org/work/](http://www.lab41.org/work/)

<comment><USER2> They've removed the part about ensembles and replaced it with a call for more "human understandable models." This at least addresses the premise of the article, though I'm not completely sold on it as a solution.

> The problem here is knowing how well the model will work with radically (or > even somewhat) different data than it was trained on. This is not the same > as doing CV or hold out.

This isn't a problem with validation techniques, but rather a problem with the data supplied to the algorithms. The established solution here is to make sure your test data actually resembles the data you intend the algorithms to work on.

Regarding the new suggestion of using models that are easy for humans to understand, I suppose my question would be "how do you know that a model that a human feels good about actually generalizes any better?" In my experience things that seem to be right by intuition are often wrong. This is one of the advantages of machine learning approaches over rules based systems, the machine can learn things a human wouldn't think of. The other big advantage is being able to scale easily, perhaps harder if you are expecting a human to manually inspect every model.

I'm definitely on board with their suggestion to continually monitor performance. There are some validation techniques that can help with that such as incremental validation.

<comment><USER3> _The established solution here is to make sure your test data actually resembles the data you intend the algorithms to work on._

That's an ideal solution, yes.

But in practice - especially for deep neural networks that take weeks to train on pretty significant hardware - most people use pre-trained models, and attempt to retrain the last layer[1].

The other issue is of course the large amounts of supervised training data needed.

[1] eg [https://www.tensorflow.org/versions/r0.9/how_tos/image_retra...](https://www.tensorflow.org/versions/r0.9/how_tos/image_retraining/index.html)

</comment></comment></comment></comment><comment><USER4> From the article: _" What I hope I have done is sufficiently piqued your interest to get you involved in Lab41."_ So this is an ad.

While I sort of agree that AI could use a bullshit meter, it's way better than it was in the 1980s. Today, much of the stuff actually works. Real work is done with AI. Deposit a handwritten check at an ATM and watch it be read properly. I'm amazed that works.

</comment><comment><USER5> Again with the use of the word AI for what is really just supervised (deep) machine learning. Pretty vacuous article on a subject covered at length since the dawn of machine learning by a very large amount of authors, in much more detail than here.

<comment><USER6> I'm curious as to what you consider AI. I tend to feel AI is a catch-all for hard problems that we don't know how to solve. Once we know how to solve something, we give it a name.

<comment><USER7> Intelligence implies the ability to reason. What people call AI are glirified calculators. And it,s getting really tiresome.

</comment></comment></comment><comment><USER8> Given who they work for, I'm surprised they open source all this - [http://www.lab41.org/work/](http://www.lab41.org/work/)

</comment><comment><USER9> Gotta love good o'le Bob G

</comment><comment><USER10> Nice try, NSA.</comment>


K-map, the weird cousin of k-anonymity (2017) - <AUTHOR> https://desfontain.es/privacy/k-map.html <comment><USER2> The article gives the hypothetical example of redacting a table row

      {zip: 85535, age: 79}

to

      {zip: 85xxx, age: 79}

on the grounds that there are fewer than k (for some tunable constant k, larger = more anonymity) 79-year-olds in zip code 85535, but many more in zip 85xxx. However. If I see the second record, because it has been redacted, I also know that whatever zip code the person actually had, there were fewer than k 79-year olds in it! This may narrow the set of candidates considerably.

So it doesn't seem sufficient to count the mere number of people a redacted row could possibly match. You have to consider the meta-level information that knowing the row had to be redacted gives the attacker.

<comment><USER3> This is true.

In fact it would be better to not indicate that the data was redacted. Instead of redacting it, change it to something else in the redacted range, preferably more common. With no hint about which pieces of data were changed, the attacker can't use what you describe.

Heck, merely including random small (unidentified) changes makes matching much, much harder.

<comment><USER4> Yes, but now when you find out that some important property you're testing for was caused by a mold outbreak in 85535, you'll be disturbed to know that the research was published for a subject in 85001.

<comment><USER5> You could publish the criteria for how your data may be jittered, but just not the specifics. Then, future users of the data could know to what extent they can rely on accuracy of the data.</comment></comment></comment></comment>


Show HN: My MVP Turn-based Fantasy RPG for iOS - <AUTHOR> http://itunes.apple.com/ca/app/sticks-n-steel/id422295527?mt=8

<comment><AUTHOR> I started coding this project at the beginning of January, using the Unity engine. I had some help with the sound, but basically I've done all the coding and art myself (some of you may be thinking "I can see that" :)

I had spent about half of December trying to decide which engine to use, and doing some exploratory coding of my own 2d animation tool. I was about 50-50 going to use Cocos2D or Unity, but I'm glad I chose Unity for the tools, and the memory management (though I'm a crusty old C++ guy).

I had initially done a version of this for the Toronto Game Jam, but it was less minimal in scope and ultimately unfinished.

So, the goal is to do frequent updates and build it out from arena combat to a more exploratory game.

Comments? Questions?</comment>


Where's Waldo? Find Him in Google Maps - <AUTHOR> https://www.blog.google/products/maps/wheres-waldo-find-him-google-maps/ <comment><USER2> I forgot how half the fun was laughing at the hysterical mini-scenes!</comment>


CSS arrow please - <AUTHOR> http://cssarrowplease.com/

<comment><USER2> I've also just converted to LessCSS mixin. Free for your use. (I added a 6th argument so that you can change the class name that gets generated).

Clickable: <https://gist.github.com/2644661>

        .arrowgasm(@position: top, @size : "4px", @background-color :  #88b7d5,
           @border-width: "2px", @border-color : #c2e1f5, @arrowClass : "arrow_box"){

             (~".@{arrowClass}") {
                position: relative;
                background: @background-color;
                border: @size solid @border-color;
            }
            (~".@{arrowClass}:after"), (~".@{arrowClass}:before") {
                    bottom: 100%;
                    border: solid transparent;
                    content: " ";
                    height: 0;
                    width: 0;
                    position: absolute;
                    pointer-events: none;
            }

            (~".@{arrowClass}:after") {
                    .b(@position, @background-color);
                    border-width: @size;
                    left: 50%;
                    margin-left: @size * -1;
            }
            (~".@{arrowClass}:before") {
                   .b(@position, @border-color);
                    border-width: @size + @border-width ;
                    left: 50%;
                    margin-left: ( @size + @border-width ) * -1;
            }
            .b(top, @border-color)   { border-bottom-color: @border-color; }
            .b(right, @border-color) { border-left-color: @border-color;   }
            .b(bottom, @border-color){ border-top-color: @border-color;    }
            .b(left, @border-color)  { border-right-color: @border-color; }
        }

<comment><USER3> You might also want to consider explicitly setting the box-sizing to content- box. I know that is the default, but if someone attempted to use this and had already adjusted the box-sizing to something else it would not work.

<comment><USER4> There are dozens of rules someone could already have on the page that would break it, why make a special case for box-sizing?

<comment><USER5> Because it's not unheard of to set * { box-sizing: border-box; }. In the same way, you shouldn't assume margin, padding or font-size to be anything in particular.

</comment></comment></comment></comment><comment><USER6> For those who enjoy sass:

        =triangle($direction: up, $size: 10px, $color: #000)
          width: 0
          height: 0
          @if $direction == up
            border-left: $size solid transparent
            border-right: $size solid transparent
            border-bottom: $size solid $color
          @else if $direction == down
            border-left: $size solid transparent
            border-right: $size solid transparent
            border-top: $size solid $color
          @else if $direction == left
            border-top: $size solid transparent
            border-bottom: $size solid transparent
            border-right: $size solid $color
          @else if $direction == right
            border-top: $size solid transparent
            border-bottom: $size solid transparent
            border-left: $size solid $color

<comment><USER7> I normally use SCSS, but this should work as well.

        =triangle($direction: top, $size: 10px, $color: #000)
          width: 0
          height: 0
          border: $size solid transparent
          border-#{$dir}-color: $color

<comment><USER8> css, sass and scss rocks. all those ideas should have their place in css3 or later.

<comment><USER9> Check out stylus, it's pretty neat too: <http://learnboost.github.com/stylus/docs/js.html>

My favorite features:

\- auto-insert vendor prefixes, even IE filters (no need for css3 'mixins', which are most of the mixins the avg dev will need)

\- _can even rasterize linear-gradient backgrounds and serve as image_ for more pre-css3 browser support. This one I haven't tested but it sounds awesome

Edit: paragraph formatting

<comment><USER6> Yeah I actually prefer stylus to sass personally, sass is just more popular (and rails default) and less magical for people that don't work with css as often.

My favorite feature actually is that it can grab properties out of parents, as demonstrated here: <https://gist.github.com/2563746>

</comment></comment></comment></comment></comment><comment><USER10> I love the creativity that results from CSS' limitations (or, arguably, its flexibility). Neat trick.

It took me a moment to realize exactly what was going on to make this work. For those interested: borders are styled in such a way that they angle toward the edges if the border is large enough:

      \ /
      / \

That gives you four "V"s to work with. Therefore, to create the arrow, you simply need to hide all but the appropriate border. The border around the arrow is accomplished by underlaying another "V".

<comment><USER11> I still don't get it. What do you mean that "borders are styled in such a way that they angle toward the edges if the border is large enough"? What causes the angle?

<comment><USER12> Take a look at this: <http://jtauber.github.com/articles/css-hexagon.html>

<comment><USER13> Thanks, I had no clue how this trick worked until I saw your link.

</comment></comment></comment></comment><comment><USER14> This is the way to do it, until the graphics guy wants the box to have a drop shadow. The pseudo elements' shadow is square, so your options are either to hope that nobody notices that the arrow doesn't have a shadow (like in this example), or to use an image for the arrow.

<comment><USER15> Here's a quick hack for a CSS arrow that casts a shadow. <http://jsfiddle.net/BKtnn/1/>

Adding border is a bit problematic but I'm sure solution could be found, just a question of few more minutes. Also, it should be doable in IE with filters.

<comment><USER14> Yes, the designer in this project naturally wanted both a border and a box shadow. I don't know how to accomplish that in CSS without using additional elements. I'm currently using strategy #1, hoping that nobody notices :)

<comment><USER16> As in the elsewhere linked CSS Hexagon tutorial (see last comment on <http://jtauber.github.com/articles/css-hexagon.html>) it's noted that there's a hexagon character. Well there are also triangle characters - which you could apply a [or multiple] shadow[s] to.

</comment></comment></comment></comment><comment><USER17> I really suck at CSS. I mean, I really suck. It takes me forever to accomplish the smallest things. I love these sites that keep popping up that generate CSS for cool little effects.

Thanks!

<comment><USER18> This and other utility and howto sites are very helpful, agreed.

BTW, to hone your CSS skills, try dropping into Firebug or Chrome's developer tools and just playing around -- very easy way to learn and get CSS stuff done. The CSS panel lets you change rules and values and see the effects immediately. I know in Chrome I use the up/down arrows on a value to slide objects and spacing around on the screen regularly. The panel also helps greatly in understanding cascading; see also the "computed styles" section.

<comment><USER19> Yup, I suck at CSS too. I still think there's something fundamentally wrong with it, but that's probably more a marker on my own ignorance than anything.

Anyway, I do have fun playing around with the FF style tools. It is quite cool and informative.

<comment><USER20> CSS was clearly designed with typography and text layout in mind - and it works wonderfully for that purpose. But it was not created flexibly enough to be well suited for anything else. You CAN make graphical interfaces with it (and we do, because we have to). But it's all hacks.

I think the browser inconsistencies are as frustrating as they are because doing seemingly "simple" things in CSS requires using advanced CSS features in boundary-pushing ways (this wonderful arrow box app is a perfect example). Browser support has to be EXCELLENT before certain simple layout things are even possible. Imagine a CSS that supported things like "vertical-align: center". That's a CSS that wouldn't NEED to have its boundaries pushed to the limits and the rough edges of browsers wouldn't rankle nearly so much.

It never ceases to amaze me how quickly people jump to defend CSS as if any criticisms imply that the author thinks we'd be better off back in the days of deeply nested tables. But the sad fact of the matter is that some things really were easier and more dependable (and in some cases, WAY more so) with tables.

<comment><USER21> > It never ceases to amaze me how quickly people jump to defend CSS as if any > criticisms imply that the author thinks we'd be better off back in the days > of deeply nested tables.

Could not agree more. Tech people are weird sometimes.

The same thing repeats itself in other contexts; for example people hate to hear criticism of git, because to many people it is the be-all-end-all of version control. But just because it's better than what came before doesn't mean there's not room to improve it even further.

</comment></comment></comment></comment></comment><comment><USER22> This is great so long as you don't still have to support IE7.

<http://css-tricks.com/browser-support-pseudo-elements/>

<comment><USER23> In IE7 you'd just have a box without the triangle, no? Isn't that graceful degradation?

<comment><USER24> Nope, you don't have a box at all (at least on this webpage). Nor the left- side box.

It looks awful on IE7 actually, the footer also degrades un-gracefully.

<comment><USER25> I wrote this a while ago, using some actual HTML elements—to get it to work super cross-browser: [http://mrcoles.com/blog/callout-box-css-border- triangles-cro...](http://mrcoles.com/blog/callout-box-css-border-triangles- cross-browser/)

</comment></comment></comment></comment><comment><USER26> That's so funny... just yesterday I built my own sass mixin to create custom triangles to do the little arrow pointer on a dialog box like this. Two triangles (border color and background color) with an offset. I toyed with using :before and/or :after to create the second background-color triangle, but the specs said you could only do text or an image so I didn't even try to make the triangle. My version required something like

. This is so much more semantic, elegant, and awesome. I love it!

<comment><USER27> I would not call this _more semantic_.

It is definitely a very cool and clean hack (more of these on HN, please), but not semantic at all: in the source there is not indication that the box is pointing at something or where it is pointing.

Looking at the domain name I thought it was a public request for a new CSS property like

      p {
          border-bottom-right-cap: solid-arrow-outward;
      }

<comment><USER26> I think it's more semantic than a non-content div or image that has no real semantic content, which is what my solution (and most I've seen) amounted to. I suppose ideally some property on the dialog container like 'data- refersto="#indicatedContent"' would be a little more semantic. And eventually that could be implemented directly in HTML and stylable in CSS. That'd be neat.

</comment></comment></comment><comment><USER28> This needs 2 easy additions, a slider to move the arrow closer to either corner of the edge it's on and a slider to change how big the arrow is.

</comment><comment><USER29> I made a bunch of this kind (and more complex) speech bubbles a couple of years ago: <http://nicolasgallagher.com/pure-css-speech-bubbles/demo/>

This is a nice little tool to help automate the production of the code for simple ones.

</comment><comment><USER30> Because the shadow seems to be setup so that the light source is at the top, when you move the arrow to any other position, the lack of shadow under the arrow becomes noticeable.

If the light source moved along with the triangle, no shadow under the arrow would be required (imo).

</comment><comment> jarek-foksa I've been writing CSS for a couple of years now and it took me like 10 minutes to understand why this is even working.

I already feel sorry for anyone that will be debugging this mess.

</comment><comment><USER31> Has a small line running through the arrow on safari (mobile). FYR

<comment><USER32> That's probably because the CSS pixel resolution isn't matching the device/screen resolution so you're seeing the edge of two elements that don't quite match (imagine a percentage being rounded). If the page included a meta viewport tag with the correct default zoom level there would probably be no line (not tested) until you zoomed in or out.

</comment></comment><comment><USER33> The harder thing is getting the boxshadow to work on that.

</comment><comment><USER34> I posted a Sass mixin for doing this here: <https://gist.github.com/2645575>

</comment><comment><USER35> Browser support? Any idea if this works in IE7/8?

<comment><USER36> No IE7/8 support because it doesn't do CSS generated content or the :before and :after pseudo-elements.

</comment></comment><comment><USER37> Arrow is repeated too many times nowadays. Neat.

<comment><USER38> I was going to use an CSS tool today, but, then i took an arrow to the knee.

</comment></comment><comment><USER39> The demo page is broken in IE8, although it does work when you copy the HTML/CSS to a new file.

<comment><USER40> _The demo page is broken in IE8_

I prefer to say that "IE8 is broken when attempting to render the demo page," but either way, good catch.

</comment></comment><comment><USER41> waste of a domain name, why do people create a whole new domain name for something as simple as this, this could have easily been a blog post! (IMO)

<comment><USER42> Not that I have anything to do with it, but two issues come to mind immediately: it may be easier to get organic search traffic for the idea, and it is a good way of keeping projects nice and separate.

</comment></comment><comment><USER43> Brilliant, I have a use case for this, nice timing :-)

</comment><comment><USER44> Thank you sir! This might come in handy sometimes.

</comment><comment><USER45> Loved the tool !!

</comment><comment><USER46> Finally.

</comment><comment><USER47> May I request that the arrow has a drop shadow too. I mean this is not exactly rocket science, but the drop shadow will be much tougher.

</comment><comment><USER48> wrap it up in a cocoa app please, so I can use it when I'm not online.

<comment><USER49> File -> Save Page As?

</comment></comment><comment><USER50> <http://news.ycombinator.com/item?id=3948850></comment>


First government survey of hikikomori - <AUTHOR> https://www.japantimes.co.jp/news/2019/03/29/national/613000-japan-aged-40-64-recluses-says-first-government-survey-hikikomori/ <comment><USER2> Funny how everyone's response is always to suggest therapy or medication and never to question whether the society people are escaping from is one worth participating in.

<comment><USER3> It's interesting what the sense of a direct translation of hikikomori implies. Original form is (if you have the font)

 _引き子守_

The first two characters imply someone who is _dragged along_ and the second two refer to a _nursemaid_ suggesting nursemaid-ing someone who doesn't want to be here. Not in defense of any such defeatist behavior but there is the question of how (and possibly why) would you impose measures to counteract that mindset if individuals and their support context are comfortable with it?

<comment><USER4> The second half of the word (the "komori" part) is not written 子守 but rather 籠り, which means "to isolate oneself".

The 引き part is less "dragged along" in this context and more "pull away from," so the literal meaning is "to pull away from (society) and isolate oneself."

</comment></comment></comment><comment><USER5> It's extremely difficult to find those around me with similar interests who actually have anything approaching 'spare time'.

Additionally, it's very difficult to be interesting without a diverse set of different friends and experiences to be the bridge between. As a 'natural leaf node' I find that I've been the first to get culled when others contract or otherwise shift their time allocations for higher priority slots in their lives.

The transaction cost of visiting friends is also affected by the size of metro areas, traffic/lack of good transit meshes, and the insane cost of housing. It often isn't worth leaving a small, opportunity isolated, corner of suburbia to meet someone else if it's going to take an hour or more one way for each trip; particularly if that's during 'rush hour' traffic hell.

Today's war for attention is insanity fueled by ad revenue and I feel like everyone's a looser in this war.

</comment><comment><USER6> Do we have any statistics from other countries?

The numbers given in the article suggest about 1% of the Japanese population are hikikomori, that seems, at first glance, quite common.

<comment><USER7> I'd imagine it would be correlated almost 1:1 with unemployment statistics, which are very well documented.

<comment><AUTHOR> There are European countries with 5%, 10%, 15% unemployment numbers. It'd be insane if they had that percentage of completely recluse population. And they obviously don't.

<comment><USER8> He said 1:1 correlated, not 1:1 equal. I.e. if for the sake of argument one in five unemployed are shut-ins but nearly no employed people are, then you have a good starting point to estimate populations.

</comment></comment></comment></comment><comment><USER9> Strange that Japan considers older recluse becoming more common than younger recluses a new development. That is the opposite of the stereotype on the West, where the "old hermit" is a longstanding archetype.

</comment><comment><USER10> I always have the "mouse utopia" experiment in the back of my mind when I read this kind of story...

<comment><USER11> I think about this too. For those who don't know the reference, I recommend reading [1].

What happened is that Dr. John Calhoun made a series of increasingly large "utopias" for mice and rats. Mice were fed, kept warm, and placed in a controlled area where there were no predators. After a bit, the mice begin to thrive and expand. But after a while things start to go astray in ways that are unexpected and a bit strange. And ultimately the population collapses and completely dies out --- even in utopia.

[1]: [https://www.atlasobscura.com/articles/the-doomed-mouse- utopi...](https://www.atlasobscura.com/articles/the-doomed-mouse-utopia-that- inspired-the-rats-of-nimh)</comment></comment>


Hobbes is a Haskell-family programming language on LLVM - <AUTHOR> https://github.com/Morgan-Stanley/hobbes <comment><USER2> Hello,

I made this project for Morgan Stanley. We've just recently open sourced it, and lately I've been working on porting it to new platforms and working with people outside of the firm to show how it can be useful in their infrastructure.

At MS, we use this to augment and implement features in low-latency trading systems (where latency budgets are very tight). We also use the storage features to efficiently record high volumes of internal trading data, which we analyze out-of-process in real-time and historically (in this use-case, it functions like a basic time-series database).

The net.H and storage.H headers here are actually themselves self-contained header-only libraries for type-safe network queries against hobbes processes and low-latency structured logging to shared memory (respectively). So with these two headers, we get a lot of processes to interact with or feed data to hobbes processes very easily.

There are some very interesting things that we're able to do with this JIT compiler and PL. I hope to write about it in more detail in the coming weeks, but the short version is that structural/algebraic data types and type classes have allowed us many creative options for analyzing and querying large volumes of trading data efficiently in real-time.</comment>


The Other Half of Science - <AUTHOR> http://www.computerscience-entrepreneurship.ml/2018/01/the-other-half-of-science.html <comment><USER2> This is a copy without attribution of a post that Josh Mitteldorf made a few days ago: [https://joshmitteldorf.scienceblog.com/2017/12/31/the- other-...](https://joshmitteldorf.scienceblog.com/2017/12/31/the-other-half- of-science/)</comment>


Show HN: Detective (a "chat" game) - <AUTHOR> http://fourtonfish.com/detective/

<comment><USER2> This is actually rather awesome. Is there a robot at any point or is one always connected to another human? As I only ever got impostors. Sadly every time I got to play impostor the detective seemed to time out.

<comment><AUTHOR> Thanks!

Unfortunately I am no longer able to add "robots" to the conversation -- see the updates on top of my article [http://blog.fourtonfish.com/post/118690105220/detective- node...](http://blog.fourtonfish.com/post/118690105220/detective-nodejs- socketio-game)

</comment></comment><comment><USER3> pretty cool! unfortunately after a bit the impostor said, "I have yo go do robot stuff now". Which kinda gave it away.

<comment><AUTHOR> Thanks!

Yeah, it's hard to make sure that people will really try to play the game "correctly", I can just give you guys a playground :-)</comment></comment>


The New Yorker: Steve Jobs's Power - A-K http://www.newyorker.com/online/blogs/newsdesk/2011/08/-the-news-that-steve.html

<comment> A-K "I've often thought of Apple as something like Singapore. It's closed, restrictive, and authoritarian. And there'd deserve to be insurrection if things just didn't work so damn well"

Witty comparison; I'd never thought about the similarities there.</comment>


First international community wireless networks - in Europe - <AUTHOR> http://wlan-si.net/en/blog/2012/10/10/we-are-internationally-wireless/

<comment><USER2> Wasn't Ljubljana already equipped with a city wide wifi network? Thought I read about it some years ago.

<comment><USER3> It still isn't. They have some antennas up, but not nearly city-wide.</comment></comment>


Coronavirus: Iran cover-up of deaths revealed by data leak - <AUTHOR> https://www.bbc.com/news/world-middle-east-53598965 <comment><USER2> I'm sorry but I'm finding it very difficult to see how this isn't straight up propaganda from the BBC.

>The number of deaths from coronavirus in Iran is nearly triple what Iran's government claims, a BBC Persian service investigation has found.

This is false. There is a very important distinction between confirmed deaths due to coronavirus, and deaths of people who may have shown some coronavirus symptoms. One is clearly an undercount, one is clearly an overcount and they aren't measuring the same thing. So to publish _on your state broadcaster_ a story deliberately mis-representing that fact is a bad move.

The data even shows that the confirmed coronavirus deaths track proportionately to suspected coronavirus deaths - actually showing that it's highly unlikely that the numbers are being manipulated.

If you go here: [https://coronavirus.data.gov.uk/](https://coronavirus.data.gov.uk/) the UK government gives the exact same measure that Iran uses as their official number.

<comment><USER3> To be fair, the BBC is about as independent as your average non-state media company, and definitely less entangled with the past five governments than anything owned by Murdoch.

That said, it is a bit awkward, given the UK's own undercounting.

<comment><USER4> > To be fair, the BBC is about as independent as your average non-state media > company, and definitely less entangled with the past five governments than > anything owned by Murdoch.

Absolutely not true. From propagating the idea that life-long advocate of equality, Jeremy Corbyn, is antisemitic, to sending well-fed journos out on boats to chase migrants (never people, migrants) around asking them what they want, the BBC is a Conservative mouth-piece.

<comment><USER5> Not to mention _literally_ portraying the chancellor as Superman. [1]

[1] [https://www.thenational.scot/news/18620765.bbc-slammed- portr...](https://www.thenational.scot/news/18620765.bbc-slammed-portraying- rishi-sunak-superman/)

<comment><USER6> And Corbyn was Lenin. I think that tells you all you need to know about the BBC.

[https://www.independent.co.uk/news/uk/home-news/jeremy- corby...](https://www.independent.co.uk/news/uk/home-news/jeremy-corbyn- newsnight-denies-photo-photoshopped-a8262351.html)

</comment></comment></comment></comment></comment><comment><USER7> They maintain two lists. Official list is whomever they checked positive for COVID-19 before or after death.

Second list which has leaked now is people who had died with suspicious symptoms but not tested for COVID-19 for whatever reason.

They admitted way before this that they believe real death toll might be 3 times higher than official numbers, and real infected might be up to 5 times higher than official numbers by comparing average/expected death rate and clinics reports.

I think this is an issue in many countries right now. The numbers are real as much as they get to test their population and everyone knows that.

<comment><USER8> It should be trivial to calculate excess deaths compared to the same period averaged over last X years, so if countries cared about transparency they could just share that, and the issue of non tested deaths would be somewhat minimized.

<comment><USER9> That statistic wouldn't be tremendously easy to interpret, since you'd have to account for the expected distribution of the differential and any secondary effects which the public health response might have caused. These effects aren't negligible and could easily produce errors similar in scale to what the BBC's calling a cover-up.

<comment><USER8> > secondary effects which the public health response might have caused

You're correct, but this is still relevant. It all depends on how the statistic is presented. Excess deaths shouldn't be presented as "deaths due to Covid-19 infection", but it could be presented as "deaths related to the Covid-19 outbreak", and people who died from cardiovascular disease because they didn't seek help due to the pandemic are still very much relevant since they are just that: _related_ to the Covid-19 outbreak.

After all, if you save 100 people in ICUs after they contracted Covid-19 that's good, but if 1 000 people die of other causes because they were scared to go to hospital then that is still relevant to understanding how the pandemic affected deaths in general, and how the response could be improved in the future compared across countries.

</comment></comment></comment></comment><comment><USER10> Looking at number of total deaths, it's possible a lot of countries are undercounting coronavirus deaths.

<comment><USER11> The NYT has a tracker for this: [https://www.nytimes.com/interactive/2020/04/21/world/coronav...](https://www.nytimes.com/interactive/2020/04/21/world/coronavirus- missing-deaths.html)

A few countries are close enough to be considered accurate. Some are off by over 100%.

<comment><USER12> Why isn't China on that list as well?

<comment><USER13> We don't have reliable information for how many people die in China per week, so we can't compare.

</comment></comment></comment></comment><comment><USER14> I appreciate the journalist's work, but I still confused if there is exist the standart way to count COVID-19 death.

> Tehran, the capital, has the highest number of deaths with 8,120 people who > died with COVID-19 or symptoms similar to it.

Not sure how Iran had formalized the similar symptoms and if they count death from another deceases that were triggered by COVID-19.

Imo the cover-up reasons seems weak.

<comment><USER15> The standard way is to count how many deaths there were in excess of what would be expected in a normal year. That way you're not relying on testing capabilities and you're not relying on how different countries define a death due to COVID. You're simply asking "how many more people have died this year than would have died normally".

Of course this will also include deaths not directly caused by infections with the virus. But it gives an overall picture of the impact of the virus and its downstream effects on a country and allows comparisons between countries.

<comment><USER16> _The standard way is to count how many deaths there were in excess of what would be expected in a normal year._

Yes, but you can only do that in aggregate over a larger period, you can't use it as a basis for daily statistics, not in the least because not all deaths are recorded the same day they occur. The daily death tallies are based on confirmed cases only, in pretty much every country.

That means underreporting of cases is the norm, not the exception [0]. I must say those graphs are a bit hard to interpret, but if I understand correctly: a gray bulge above the dark gray dotted line implies excess deaths not officially attributed to Covid-19. From those figures, it seems only Denmark and Germany are accurately reporting Covid deaths, and Belgium, France and the US may be overreporting the death tally (although it can also mean that not all deaths in the past weeks have been recorded yet in the official figures).

[0] [https://www.theguardian.com/world/ng- interactive/2020/may/29...](https://www.theguardian.com/world/ng- interactive/2020/may/29/excess-deaths-uk-has-one-highest-levels-europe)

<comment><USER17> The excess death numbers in the US are from the CDC, and their numbers are incomplete unless you go back about 8 weeks. I don't know why people keep creating graphs and charts from their incomplete data! It paints a wrong picture every single time.

<comment><USER18> Creating a graph with only data from 8 weeks ago would be an accurate graph, but it would tell us what we should have been doing 8 weeks ago, not what we should be doing now. Of course, death is already a lagging indicator, so really this would tell us what we should have done 10 weeks ago. It's just not an actionable graph, because it's too delayed.

Of course, using incomplete data may not tell us anything either. It's also not an actionable graph, because there's too much uncertainty.

</comment></comment></comment></comment></comment><comment><USER19> Russian cover up is still waiting to be exposed.</comment>


Python and Django on Heroku - <AUTHOR> http://blog.heroku.com/archives/2011/9/28/python_and_django/ <comment><USER2> Python 2.7? They're _years_ ahead of Google App Engine!

 _Addendum:_ The previous statement is slightly tongue-in-cheek, but: (1) GAE is running a version of Python (2.5) so old, that it's hard to get fully patched binaries of it anymore (depending on your platform); and (2) By "years ahead," I don't mean it will take Google years to catch up, but merely that their Python version is years old. Puzzling, given that Guido works for Google and has for years.

 _Second Addendum:_ Linked: an issue opened in 2008 pleading Google to add 2.6 support. Three whole years ago. (Ironically, perhaps, the issue was closed as a duplicate of a 2010 issue asking for 2.7 support.) [http://code.google.com/p/googleappengine/issues/detail?id=75...](http://code.google.com/p/googleappengine/issues/detail?id=757)

<comment><USER3> App Engine announced they will release a Python 2.7 runtime in several months. It's currently in their trusted tester program.

<comment><USER4> More details on 2.7 and GAE: <https://sites.google.com/site/gaepython27testing/>

</comment></comment></comment><comment><USER5> As much as I've come to enjoy and appreciate the various start-ups whose mantra was "We're Heroku with Python/Django capabilities", it will be interesting to see which of those survive the next 8-12 months now that Heroku officially supports that stack.

It'd be a shame to see ep.io and Gondor go the way of the dodo bird, but what sets them apart now?

<comment><USER6> Epio cofounder here - we're not worried by this, and we've known it was coming for ages now.

The hosting market is plenty big enough for more than one player - we're expanding to multiple languages, much like Heroku expanded from Ruby, which provides an ample feeding ground of old-style hosts to slowly steal business from - and there's also still a lot of room for innovation.

Heroku, as much as I like their product - and I do, which is why we started this a year ago - still has its flaws and idiosyncracies, some of which are those low-level-design type of choices where if you go the other route, a _different_ set of people complain. You can't be all things to all people, and I don't think anyone will end up being that.

<comment><USER7> "some of which are those low-level-design type of choices where if you go the other route, a different set of people complain"

This is actually a wonderful scenario. If either party successfully grows the market, both can benefit.

</comment></comment></comment><comment><USER8> I loved Heroku, many moons ago when I was working in Rails. As I've emigrated away from Rails to Django, I've found dotcloud to be the premiere platform -- I want to qualify this, it is the premiere polyglot platform, but for each individual environment I've deployed on dotcloud, their experience has been the best.

I haven't used dotcloud's Ruby/Rails stack, so I can't compare that, but Heroku is definitely fighting a hard battle if they're going to swing me from dotcloud, but it's always good to have competition, and if anybody is going to bring their A-game, it will be Heroku, who were sort of pioneers in the space.

The Heroku Python Free platform might be better in some instances than dotcloud's, and I'll definitely investigate that, but for anything I can think of using, dotcloud has been amazing for me.

<comment><USER9> I still find Dotcloud's pricing violently expensive (and I have a free VIP plan). YMMV I guess.

<comment><USER10> Still:

       Q. Are there discounts for education and non profit users?

       A. Yes. DotCloud provides special free and low cost plans for open source
       developers, educational users and other qualifying non-profits. To inquire
       about adding free or low cost capacity to your DotCloud account, please
       contact support.

</comment></comment></comment><comment><USER11> I've been using Flask on Heroku since the Facebook apps announcement, and set up a boilerplate template here: <https://github.com/zachwill/flask_heroku>

</comment><comment><USER12> Heroku's cedar stack can now detect apps using:

      * Ruby
      * Node.js
      * Clojure
      * Python
      * Go ??
      * Scala
      * PHP
      * Java
      * Perl ??

Anything missing here? That covers a lot of whats out there.

<comment><USER13> Logo. [http://blog.heroku.com/archives/2011/4/1/announcing_heroku_f...](http://blog.heroku.com/archives/2011/4/1/announcing_heroku_for_logo/)

</comment></comment><comment><USER14> This is great! Looking at the Django tutorial in the docs [1], I think it's too bad, though, that they don't provide a build-in, well tuned WSGI Server. This way, you have to choose your own WSGI server, configure and update it yourself.

[1] [http://devcenter.heroku.com/articles/django#using_a_differen...](http://devcenter.heroku.com/articles/django#using_a_different_wsgi_server)

<comment><AUTHOR> We do not automatically use a wsgi server because we want to offer flexibility to our users. Additionally we felt that having less magic in what we automatically do to your application (other than run it) was more of the Python way.

<comment><USER15> > "was more of the Python way."

That you tailor each environment to the language/framework around it is a great touch.

</comment></comment></comment><comment><USER16> $ ls

app.py,requirements.txt,Procfile

$ git init

$ git add .

$ git commit -m "Init"

$ heroku create --stack cedar

$ git push heroku master

With that Heroku becomes the Heroku for Django.

<comment><USER17> $ epio upload -a appname. epio is epio for Django.

</comment></comment><comment><USER18> Anyone know if there's a self-contained way to spool up a dyno sporadically to complete scheduled tasks?

I have a webapp now that's hosted on a VPS and uses Celery to schedule tasks. The tasks themselves only take about a second, the dyno needs to stay active for ~5 minutes to service a bunch of HTTP requests from another webservice that will result from the task, and then shut down until the next task. There are O(100) tasks spaced throughout the day, and they each must be completed at a very specific time. My aggregate dyno-hour requirements are very low, but I haven't figured out a way for a dyno to turn itself on and off for scheduled tasks this way. Admittedly, my use case is a bit niche, but a solution sure would be useful. Whiteboxing my webapp in such a way that it could be deployed on Heroku would be great, but keeping a dyno running all month to run the Celery polling process, when 'actual' computing is happening << 1 percent of the time, is a bit steep :)

<comment><USER19> Use a single dyno, and set the caller's HTTP timeout threshold to a high enough point that it can deal with the delay associated with reallocating your app.

</comment></comment><comment><USER20> Some time ago Ask The Pony blog made a detailed tutorial on deploying Django on Heroku and how to make it perform 8 times faster than usual using some nice tricks: [http://www.askthepony.com/blog/2011/07/getting-django-on- her...](http://www.askthepony.com/blog/2011/07/getting-django-on-heroku- prancing-8-times-faster/)

</comment><comment><USER21> Interesting... I wonder if this makes Google App Engine more appropriate for internal apps for Google Apps customers? For quick/easy personal hacking/development having a Python stack on Heroku seems pretty attractive now.

<comment><USER22> Personally I don't like Google App Engine, but I can understand its value -- it has reasonable free quotas (even now, after the pricing scheme changed), and your app ends up running on Google's infrastructure which is rock solid.

However, I never understood why people like Heroku ... it's like renting instances on EC2, only 10 times more expensive.

Of course, people then start enumerating a whole bunch of stuff that they don't have to worry about when using Heroku. However, when starting out, configuring a server is just like configuring your localhost development environment. You just have to start with something that works, then gradually keep learning.

Heroku doesn't provide anything to me other than an unreasonable free quota that's basically useless for serving anything other than static content (poorly) ... even GitHub does a better job with their static pages option, since GitHub's servers don't go sleeping when unused.

<comment><USER9> "it's like renting instances on EC2"

False. I'd suggest you take a look at their architecture overview. It's not like renting instances on EC2 at all because dynos != virtual machines. <http://www.heroku.com/how>

"Of course, people then start enumerating a whole bunch of stuff that they don't have to worry about when using Heroku. However, when starting out, configuring a server is just like configuring your localhost development environment. You just have to start with something that works, then gradually keep learning."

This may be fine for some, but for those that _don't_ want to worry about sysadmin work and want to focus on rapid iteration, Heroku makes more sense. Besides, it's not the initial configuration that is the most painful, it's the maintenance and cost associated with managing your own virtual instances, and infrastructure pieces such as reverse proxies, caches, etc.

"since GitHub's servers don't go sleeping when unused."

I'm not even sure what this means. Can you elaborate?

<comment><USER22>
          dynos != virtual machines

Well, yeah, a small instance on EC2 is the equivalent of 20 dynos, maybe more.

What you do get with dynos is scaling out when you need it, however you can do the same thing by having a prepared AMI, a load-balancer and a bunch of scripts with which you can start new instances in seconds.

         focus on rapid iteration

I really do think that sysadmin work and rapid iteration are orthogonal. When you're starting out, administrating you servers is something that hardly takes up any time ... but the flexibility is priceless.

         ... infrastructure pieces such as reverse
         proxies, caches, etc.

Well, Cedar doesn't have Varnish anymore -- and IMHO, setting up Varnish is just a day's work, which includes configuring it for your own needs.

Yes, Heroku takes that away by (1) giving you a useless Varnish configuration OR (2) eliminating Varnish altogether, as they couldn't figure out a common denominator.

Also, I know that "reverse proxies" and "caches" sound bad ass, but it's really a solved problem.

Of course, infrastructure can get very hairy further down the road, but that's what I've been saying -- when starting out, you just need Passenger or mod_wsgi, as in one "sudo aptitude install" or "gem install" away. It took me a day's work to configure an EC2 instance, including a deployment workflow with Capistrano (for the first time ever). That server is still running just fine, with no further maintenance.

And if small deployments is not Heroku's strength, than what is? If you've got a successful app that needs special infrastructure care and you can't afford a good developer/sysadmin to take care of it, then you're doing it wrong.

<comment><USER23> > you can do the same thing by having .... and a bunch of scripts with which > you can start new instances in seconds.

Right, so if you already have everything you need, you don't need to buy it. This argument is similar to "I don't need water when I'm not thirsty". Hardly compelling.

> administrating you servers is something that hardly takes up any time

Sure if your servers never get any traffic, or you never scale.

> It took me a day's work to configure an EC2 instance, including a deployment > workflow with Capistrano (for the first time ever). That server is still > running just fine, with no further maintenance.

Right, it took me less than a minute to do that with heroku, and with less dependencies (on capistrano). When I decide my app needs an extension written in node, that is another minute on deployment for me, and another day for you. I'd gladly compete on those terms :)

> And if small deployments is not Heroku's strength, than what is? If you've > got a successful app that needs special infrastructure care and you can't > afford a good developer/sysadmin to take care of it, then you're doing it > wrong.

This is just your highly opinionated view of things. Who ever said anything about 'special infrastructure'... Heroku deal in commodity infrastructure (that's the point). You also never successfully made the point that small deployments are not their strength.

</comment></comment></comment></comment></comment><comment><USER24> Awesome news topped with even more awesomeness that Heroku is continuing to truck along post-acquisition.

We're now looking into +/-'s of moving from RAX Cloud Servers to Heroku.

</comment><comment><USER25> I'm currently developing django app on a ec2 instance. What's the advantage of heroku over just plain ec2?

<comment><USER11> Ease of use — especially for prototyping ideas. It's easy to push a git repo to heroku and have it fire up your site — but if you've already got a fabric script to do the same thing with EC2, then that might be the better approach for you.

</comment></comment><comment><USER26> This is pretty much what I've been waiting for to finally migrate my project to Heroku. I know you've been able to _unofficially_ run Python/Django on the stack for a while now, but a lack official support (even in beta form) was all that had been keeping me from taking the leap.

</comment><comment><USER27> Python minimizes magic and maintains backwards-compatibility? LOL.

(I'm a programmer who recent dived into python - it's awesome and it's easily my favourite language to use now - but those statements are fallacies).

<comment><USER28> Compared to Ruby on Rails, there's less magic going on.

</comment></comment><comment><USER29> Does anyone know if the stack will only support WSGI or could I also run a Tornado instance on the cedar stack? I know when I tried dotcloud several months ago it only supported WSGI.

</comment><comment><USER30> You always have to work with the information available at the time, and personal tolerances for risk, but Heroku has become a case study in selling too early.

</comment><comment><USER31> Can Haskell be next?

</comment><comment><USER32> Great news! But why is the example for Flask?

<comment><USER33> Maybe because it takes less code to write a "hello world" with Flask than with Django.

<comment> paul-woolcock Definitely this. A database is a requirement for a Django app, but not for Flask. I always seem to go back and forth about whether to start a micro- framework like Flask and use plugins and Python libs to assembly a framework that has only what I need, or use Django and (possible) include a lot of code that will never get run.

<comment><USER5> It really depends on your vision for the app. If this is a single weekend project or a toy web site that you'll likely never touch again, or if you do will only be for minor updates, then I say go for Flask. It'll get in your way less and let you just launch it sooner.

For anything larger than that though, Flask has always turned into a giant headache for me. As soon as I ever start to try and organize the project better because it's become too large for a single file, all shit hits the fan. Supposedly this has gotten slightly better with 0.7's Blueprints, but I've yet to see it. Circular imports and Flask's global variables bite me in the ass on a regular basis. I also have come to really miss tons of features that Django offered, like URLconfs. The decorator technique works great when you can count the number of URLs that your app has on two hands. Beyond that, ack/grepping my project to find where anything lives isn't my idea of maintainable. With Django I can very cleanly trace through the whole request/response cycle in my head and by looking at the code paths.

<comment><USER34> I've been running multi file apps since before 0.7. If you read the documentation or searched mailing lists or looked at example applications you can see how. Blueprints made it even better.

2\. In the documentation you will find that you can list out routes just like django. <http://flask.pocoo.org/docs/api/#flask.Flask.add_url_rule>

<comment><USER5> I've read through all the documentation, and researched extensively. I still ran into problems way too often to ever warrant using it over a full framework for larger projects.

<comment><USER34> I'm not trying to get you to use Flask or anything. But I definitely find it interesting that you have "read through the documentation or researched extensively" but were unable to find `add_url_rule`.

Either way, I'm glad we have selection and choice. At least we both like python! Peace

</comment></comment></comment></comment></comment></comment></comment><comment><USER35> Django has been able to run on Heroku for a while now. This topic keeps coming up over and over again and it is very old news. There are a lot of major flaws in running Django on Heroku right now too simply because of how their system works.

<comment><USER36> Hi, care to list some of the flaws?</comment></comment>


2009 May Smile on Disruptive Startups - <AUTHOR> http://gigaom.com/2009/01/03/2009-may-smile-on-disruptive-startups/ <comment><USER2> This quote from Richard Moross, CEO of Moo, on how to assess business ideas in 2009 was interesting:

       Is this really a business? Is it really different? If the elevator pitch includes the words 'Twitter',
       'social network' or 'it's web app X meets geek meme Y' you probably need to rethink things - those
       days are gone. Today your idea needs to be super-relevant: do people actually need this, or are you
       just a solution looking for a problem?

Full interview here - [http://www.guardian.co.uk/media/pda/2008/dec/25/digitalmedia...](http://www.guardian.co.uk/media/pda/2008/dec/25/digitalmedia- downturn)

<comment><USER3> Hasn't this ALWAYS been the case? It's just that every now and then, heads slowly sink up asses and caution is thrown to the wind.

It's never a bad time if you really DO have a good idea.

</comment></comment><comment><USER4> Summary: Create value.

What a novel idea...

<comment><USER5> It's good to hear people talking about creating value. Hopefully the days of useless social networking, sheep throwing, farting phone vc fueled exit strategy bullshit will be over for a while (until the next bubble).</comment></comment>


New EFF Documentary: Flying An Airship over NSA's Utah Data Center - <AUTHOR> https://www.youtube-nocookie.com/embed/EsEkmHRbThk?rel=0

<comment><USER2> I was in the airship and shot some of this footage. Nothing formal here, but feel free to ask me anything :)

<comment><USER3> Any repercussions?

<comment><USER2> Nope. We had checked in advance about the legal airspace and stuff, so we didn't expect too much blowback.

<comment><AUTHOR> Thanks for all you guys are doing.

</comment></comment></comment></comment><comment><USER4> At 1:36 , they use "suspicionless surveillance".

Given how many laws there are in this country, everybody is guilty of something. And it's probably a felony, and tremendously inane.

</comment><comment><USER5> We must destroy freedom to save freedom.

<comment><USER6> The beatings will continue until morale improves.

</comment></comment><comment><USER7> timestamp 0:07 - s/facilty/faculty/</comment>