SIGGRAPH: Rigid-Body Fracture Sound Experiments - <AUTHOR> http://www.cs.cornell.edu/projects/FractureSound/

<comment><USER2> I forsee a Half-Life 2 mod consisting solely of people shooting dinner plates through glass windows.</comment>


For sysadmins, the times are a-changin' - <AUTHOR> http://devopsanywhere.blogspot.com/2011/10/what-is-devopsanywhere.html

<comment><USER2> The devops movement, while interesting, seems to be orbiting around Ruby... Are there any tools language agnostic or other-language-than-Ruby oriented?</comment>


Usql – A universal command-line interface for SQL databases - <AUTHOR> https://github.com/knq/usql <comment><USER2> I'm the author of pgcli and mycli. I was tempted to write a universal cli that supported both postgres and mysql, but then I decided to keep them separate for the following reasons:

1\. Psql comes with backslash commands that are implemented at the client level and not in the database. So you can't simply send `\dt` to the database. That will fail. The same is true for mysql as well (such as `\u`).

2\. There wasn't a huge overlap in users who use Postgres and Mysql at the same time. Even for the minority that do, I tried to keep pgcli and mycli behave similarly so they can switch between them seamlessly.

3\. The code complexity needed to support multiple database backends would make it hard for a newcomer to contribute to the project. This is just my guess and I don't have data to back this up.

4\. I didn't think I could do a good enough job of maintaining parity with psql and mysql's default client if I had a universal client. I still don't have complete parity but it is close enough that users don't feel like their compromising a lot.

I'm happy you're taking on this challenge and I'll be glad to give you pointers on how the sql completion engine works in pgcli and mycli.

Cheers!

<comment><USER3> Man, what I really would love is some ipython console with SQLAlchemy stuff built in that auto-sets up the connections and session objects, and that has convenience functions for table / database info and dumping data out to CSV/TSV - that'd come close, I think.

<comment><USER2> You know this actually exists: [https://github.com/catherinedevlin/ipython- sql](https://github.com/catherinedevlin/ipython-sql)

I found out about this from one of the core-devs of pgcli. He made it possible to embed pgcli inside ipython. We wrote a blog post about how to use this: [http://pgcli.com/embedding-pgcli-in-ipython.html](http://pgcli.com/embedding- pgcli-in-ipython.html)

<comment><USER3> Wow, I even had this repo starred but somehow ignored the contents entirely. Thank you, kind stranger! This'll be so useful to me.

</comment></comment></comment></comment><comment><AUTHOR> This is a new tool (literally only started in the last 24 hours), but already provides a common, usable interface to PostgreSQL, MySQL, Microsoft SQL Server, SQLite3, and Oracle databases. Plans to add support for other NewSQL databases (VoltDB, etc) in the future, as well as to make it have feature parity (and the same interface as) postgres' psql command line tool.

<comment><USER4> My sincere thanks for your work on xo. This project looks promising as well.

</comment></comment><comment><USER5> Is there a way to try this without figuring out how to set up a go build environment? I've never used go, but wanted to try this out.

I installed go 1.8, and then issued the prescribed command line. It tells me `# github.com/mattn/go-sqlite3 exec: "gcc": executable file not found in %PATH%`. This looks like a rabbit hole of unknown depth that I'd prefer to stay out of.

<comment><AUTHOR> This project is only roughly a day and a half old. However, I will make binary releases available in the future.

</comment></comment><comment><USER6> Might run into some copyright/trademark issues [https://docs.microsoft.com/en- us/azure/data-lake-analytics/d...](https://docs.microsoft.com/en- us/azure/data-lake-analytics/data-lake-analytics-u-sql-get-started)

<comment><USER7> Is it even possible to copyright/trademark that when 75% of the name is a technology they do not own?

</comment></comment><comment><USER8> This is an ambitious goal since the supported databases are so different. Postgres supports window functions for example whereas SQLite does not.

<comment><USER9> Is the goal of this project to homogenize syntax or homogenize the call level interface (api)? I can understand the need for the latter, not the former.

<comment><AUTHOR> This is not about making a generic SQL implementation. This is simply about being able to execute SQL consistently, and to use the same CLI interface for doing so. The user would still need to know the individual SQL dialects for each database they are using. However, at least then if you become familiar with, let's say, mysql by using this tool, the same configuration/settings/etc. will be used for all the other databases.

<comment><USER10> Oh I see - so it is an adapter not an abstractor. I'm curious what the advantages will be over the widely supported ODBC API. Will you add ODBC support?

<comment><AUTHOR> I've already looked into adding ODBC support, and will try to add it. The real limitation here is that I am not working with a ODBC setup locally, so writing/testing anything for it is not really possible at the moment. Also, I believe all the ODBC Go driver implementations require the use of a C lib/dll, so it would need to be similar to the Oracle support and not "out of the box". The beauty of this, is that at the moment you can simply do a 'go get -u github.com/knq/usql' and have a working SQL client that is exactly the same for the 4 "natively supported" databases, and with Oracle support if you have the right local client libraries.

</comment></comment></comment></comment></comment><comment><USER11> Reminds me of sqlline (used it like 8 years ago, not sure if still active). A Java based SQL command line client. [http://sqlline.sourceforge.net](http://sqlline.sourceforge.net)

<comment><USER12> Came here to mention this. It is the superior solution because it will take an arbitrary JDBC driver thereby supporting far more databases.

</comment></comment><comment><USER13> It's a shame no-one has used DBI and SQL::Abstract etc to build something similar in Perl.

<comment><USER14> see GNU sql, it comes with GNU parallel [https://www.gnu.org/software/parallel/](https://www.gnu.org/software/parallel/) "GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.

The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.

When using GNU SQL for a publication please cite:

O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.

<comment><USER15> Ah, it's that GPL + nag/guilt-trip model again...

Following that rationale, why doesn't any sane person cite Matlab, R, Tex or MS Word on their publications?

</comment></comment></comment><comment><USER16> This is pretty neat!

What would really make a difference in how we perceive the SQL CLI world would be an IPython-like interface – complete with syntax highlighting, tab- autocomplete menus, easy multiline backlog entry editing etc.

<comment><AUTHOR> Yes, I already have a working branch with syntax highlighting and autocomplete hacked together, but the issue is that it breaks for some of the databases. When I can get it stable, I'll make available publicly.

</comment></comment><comment><USER17> A few years back I toyed with the idea of building something like this. It was a dark time that I was stuck using a database besides Postgres.

If you want to really make this usable here's what you need to add:

\- usql -f foo.sql

\- Allow @path/to/file.sql and @@path/to/relative/file.sql references

\- Common syntax for referencing env variables in SQL (each DB's utilities have slightly different syntaxes so have to pick one)

\- External transaction demarcation to ensure a set of commands get executed together (ex: psql --single-transaction)

\- _Some_ autocompletion (if you plan on this being used for interactive use)

<comment><AUTHOR> Auto-completion will come in time. I don't know if I would allow "relative" pathing via @@, I'm not sure why you wouldn't just use general paths. As for the other options, such as -f, etc., yes those will be put in place. I am hoping to make this a 100% drop in replacement for psql, so whatever psql supports on the command line, as well as the various backslash commands (ie, \f) will be added.

<comment><USER17> Relative paths are a must. I don't think I ever use non-relative paths.

Say you have a folder structure like this:

        sql/
          tables/
            foo.sql
            bar.sql
          main.sql

And the contents of main.sql are something like this:

          @@tables/foo.sql
          @@tables/foo.sql

Then you can run main.sql via: usql -f sql/main.sql

If script file resolution is not relative pathed to the parent, it gets very fragile as you must invoke the parent from a specific location, or, you have to hard code full pathes in your scripts (arguably even uglier).

<comment><AUTHOR> This is a feature in psql? I must admit if it is, I'm not familiar with it.

<comment><USER17> Yes psql supports it. The syntax is "\i path/to/file.sql" or "\ir relative/path/to/file.sql". The latter is resolved relative to the parent script. Makes nesting scripts possible without losing your sanity.

<comment><AUTHOR> Yes, if it's in psql, I'll get it into usql in the near future.

(for certain large values of near).

</comment></comment></comment></comment></comment></comment><comment><USER18> Slightly off topic, but I was recently looking for a tool like mytop but for Postgresql. Does any one know if it exists?

</comment><comment><USER19> Oracle sqlcl and sqldeveloper has/had multiple SQL support. You have to use a hint on occasion and it is biassed

<comment><USER19> Biased towards oracle SQL. I can check it out it was mostly for migration to Oracle Database.

<comment><USER19> Usql requires go. Sqlcl is currently 12mb.

<comment><USER20> It doesn't require Go any more than nginx requires C. Go executables are for the most part static, and depend on very little, making them highly portable across distros.

</comment></comment></comment></comment><comment><USER21> Is this just running Go's database/sql with various adapters?

If it is that you can even include Neo4j support.

<comment><AUTHOR> Yes, the plan is to add support for any database (relational or otherwise) that has the ability to be controlled through a "command and query" style interface. This project is only about 1 day old at this point, but I'll make a point to rapidly integrate as many other DBs that are supported by Go as fast as humanly possible.

Btw, it would be relatively straightforward to add support on your own. PRs always are welcome!

</comment></comment><comment><USER22> Somewhat confusing since azure data lake analytics uses a sql-esque language called u-sql

<comment><USER23> Even searching for usql on GitHub [1] returns an Azure U-SQL related result first.

[1]: [https://github.com/search?utf8=%E2%9C%93&q=usql](https://github.com/search?utf8=%E2%9C%93&q=usql)

</comment></comment><comment><USER10> Is this like ODBC but with its own SQL dialect to abstract away the underlying dialect?

<comment><AUTHOR> No. You need to use the native SQL dialects for each database. It provides a standard readline-style interface and uses various Go SQL database drivers to talk to the actual databases.

</comment></comment><comment><USER24> I can see this being a very handy tool! Kudos on the work being done here!</comment>


Trump Accuses Amazon of Not Paying 'Internet Taxes,' Which Aren't a Thing - <AUTHOR> http://gizmodo.com/trump-accuses-amazon-of-not-paying-internet-taxes-wh-1796483336 <comment><USER2> We'd love to know if Trump pays taxes but he's far to evolved to release his tax returns like other Presidents and candidates.

I won't even comment on his complete lack of ability to be a President...</comment>


Open Source Modular Coil Gun - <AUTHOR> http://www.youtube.com/watch?v=KeSjeUIltcI A friend of mine has designed and built a coil gun.  The design is open source.  You can download and (3D) print the modules yourself.  Pretty cool, eh? <comment><AUTHOR> Modular gauss gun with optical trigger. The design can be downloaded from Thingiverse <http://www.thingiverse.com/thing:23539></comment>


My Raspberry Pi cluster - <AUTHOR> http://blog.erratasec.com/2016/07/my-raspeberry-pi-cluster.html <comment><USER2> I'm not OP but I've done this stuff on and off for two decades and I thought I'd toss out some answers.

About 20 years ago I got started using a dozen or so surplus Pentium75 desktops and by combining the working parts and best/largest parts I ended up with eight or so nodes with a reasonable amount of memory and local storage.

A write up would have been pointless it was pretty much standard like you'd read in any linux magazine of the era.

I used one of the PBS options for batching and some homemade stuff to rip audio CDs to mp3 in a distributed manner and completely unproductively fooled around with PVM and MPI. In the old days PVM was stable and easy to use although MPI was predicted to be the wave of the future. In fact today PVM is pretty much dead and everyone uses MPI or the ever popular NIH homemade stuff. "Back in the old days" we did things like spawn off ray tracing jobs semi- manually with bash scripts and perl scripts. You'd write perl that output animated povray files given a frame number and then queue up like 120 ray tracing jobs and then stack them into video and thats how you got a minute of bad ray traced animated movies. Not being much of an artist my animated ray tracing was limited to bad physics demos like mirrored ball bouncing. As most other cluster ops do I calculated a lot of "X+1" and "Is X a prime" to test. Like most screwing around, honestly I just turned a lot of KWh into heat.

The wife acceptance factor was incredibly low. Although admittedly I'm still married, to the same awesome woman even. Eventually I got rid of all the machines. But for a year or two it was a lot of fun. I thought the noise level was bad... then I replaced all the boxes with dell towers a few years later that sounded like jet engines. Then IBM 1U rackmount servers P4 surplus around 2010-ish which did a pretty awesome job of running LXC before containerization became "cool". Portability ha ha ha the size of a closet in the middle of my home office, looks like an organized disaster zone. Really looking forward to building a pi cluster to screw around with that'll fit in a shoebox with room to spare. OMG I could build a toy cluster so small I could drop it into my desk drawer when I'm not playing... Oh if I had more time this summer I'm feeling the urge to build another cluster...

Power doesn't matter as long as the circuit breaker doesn't pop. At one point I figured I was burning 1100 watts using the dell towers (and yes the room does get warm) which implies running 24x7 would cost me $1100/yr (which is very cheap compared to a training class or many other hobbies, but I digress) however I suspect I never powered up that cluster more than maybe 300 hours. I spent 10x more money on a then new technology 100 meg ethernet switch than I did on the power the system used. Back in the dinosaur era we mostly used hubs which were unswitched one big broadcast region, probably sounds very weird to modern young kids.

I can't even begin to list all the stuff I learned. The jump from admin'ing one unix box to admin'ing a cluster is almost as big as the jump from being an appliance user to being a cluster admin.

What I didn't learn was the scalability problems of having 10000 nodes, having less than ten generally, I did a lot of setup by hand. Although I did all the OS level stuff automation like homemade scripts, puppet, etc. A lot of cluster admin is related to the "fun" of physical hardware support for 10000 individual boxes and I miss out on that. Also running 24x7 how do you handle dusty air filters or whatever at that scale for day to day operation? Power failure recovery must be non-trivial.

Its hard to say "well I'd never have gotten job X except for..." because I might have anyway. As a hobby its been tangentially tied in with my day job on and off since at least the turn of the century.

<comment><USER3> I did the exact same thing (cluster to compress MP3's), only I had a paying customer for the job which (handsomely) paid for the work _and_ the cluster.

The boxes are long gone but the pictures remain:

[http://www.clustercompute.com/](http://www.clustercompute.com/)

<comment><USER4> I like how it was called "beowulf cluster" back then, it's a while I read that name. We had an 160 node beowulf cluster (had already USB support, no floppy drives) for OpenMP and CILK jobs it was amazing. [https://en.wikipedia.org/wiki/Beowulf_cluster](https://en.wikipedia.org/wiki/Beowulf_cluster)

</comment></comment></comment><comment><USER5> [https://pi-hole.net/](https://pi-hole.net/) is my favorite pi software, it's network wide DNS server for ad blocking which I supplement with a bunch of privacy/security lists from [https://github.com/StevenBlack/hosts](https://github.com/StevenBlack/hosts). 5% of my traffic was blocked today which is heaps considering it's counting mountains of requests to eg Dropbox's API.

It even ran powered straight off my router's usb port, although I just put in a VM now: [https://github.com/benlowry/pihole-extended- hosts](https://github.com/benlowry/pihole-extended-hosts)

<comment><USER6> I really like it too but after extended testing I found it was incompatible with NoScript's Application Boundaries Enforcer because pi-hole replaces ad requests with an empty HTML 'page', rather than simply dropping or blocking ads like a browser ad-blocker would.

I don't know enough about how this works to figure out how to work around it so I reverted to blocking ads at the router with a hosts file. NoScript provides too many other benefits so I had to keep it and stop using pi-hole.

<comment><USER7> Do you actually use the ABE specifically, that part can be turned off (NoScript > Options > Advanced : ABE and then uncheck the enable checkbox) without stopping any of the other aspects from working.

I'm not a dev but looking very quickly at pihole it seems it serves an index.html file ([https://pi-hole.net/faq/is-it-possible-to-change-the- blank-p...](https://pi-hole.net/faq/is-it-possible-to-change-the-blank-page- that-takes-place-of-the-ads-to-something-else/)) in place of ad content, you might make that a link to /dev/null or just make it a blank file, and see if that works? I suspect it won't as the ABE is checking for the correct origin of content and pihole is spoofing the origin.

<comment><USER6> I use ABE only because it's enabled by default and it seems like a sensible thing to do. At the time I tested pi-hole, reverting to router hosts blocking seemed easier so I left it at that. Maybe I'll look at it again and see if I can selectively disable ABE for the pi-hole's IP address which should solve the issue.

<comment><USER7> Hmm, I think ABE works back-to-front of what you want, eg [http://stackoverflow.com/questions/20111530/noscript-abe- all...](http://stackoverflow.com/questions/20111530/noscript-abe-allow-a- remote-site-to-access-local-resources). You let a remote page have access to local data, so you'd need access rules for all the external pages that are adblocked to let them have access to the local content that pihole wants to stuff the ad spaces with.

This might work, using [https://noscript.net/abe/abe_rules.pdf](https://noscript.net/abe/abe_rules.pdf), but I'd ask at the NoScript forums. The rules cascade so it would need to go at the top.

        #ABE rules
        # ALL matches any URI
        # accept lets it happen, sandbox and anon probably would be fine too
        # INCLUSION is any sub-page level part, like images and such, can specify further which inclusions
        # LOCAL is any local address including those in noscript.ABE.localExtras (see about:config);
        Site ALL
        Accept INCLUSION from LOCAL
        # Deny # probably only needs that if you're not prepending already made rules??

That might negate much of ABE though as you're letting external sites reference anything that they find on the local domain.

Let me know if it works!

</comment></comment></comment></comment></comment><comment><USER8> I'm still hoping for someone to build a SBC like the RPi that takes POE for power. USB is convenient for general use, sure, but for something like this cluster it would be ideal to have a single wire in and out.

Unfortunately it's probably a chicken-and-egg thing, where POE is too expensive until a lot of boards support it, and boards won't support it because it's too expensive.

<comment><USER9> I'm not sure that cost is the biggest problem. PoE splitters with USB ports can be had for ~$20. Though, relative to an RPi, I guess that is somewhat expensive.

Ex: [https://www.amazon.com/gp/aw/d/B019BLMWY0](https://www.amazon.com/gp/aw/d/B019BLMWY0) (note, I haven't actually used this product).

A quick ebay search shows some that are ~$5, so I'd imagine it wouldn't be all that hard for an experienced board developer to get the price lower at higher production rates.

<comment><USER10> Here's another one that's a brand you may have heard of and it also does multiple voltages too. Only $15. Can even get it as a kit if you don't have a Poe switch.

[https://www.amazon.com/gp/aw/d/B003CFATQK/ref=mp_s_a_1_1?ie=...](https://www.amazon.com/gp/aw/d/B003CFATQK/ref=mp_s_a_1_1?ie=UTF8&qid=1469245044&sr=8-1&pi=SL75_QL70&keywords=poe+injector&th=1&psc=1)

</comment></comment></comment><comment><USER11> Cheap maybe, but if you really use Raspberry Pi a lot, you will find that Pi isn't that robust after all. I installed 20 of them two years ago, so far at least replaced 4 of them during the time. Can't comment on the recent models though.

<comment><USER12> I've been hosting [http://www.pidramble.com/](http://www.pidramble.com/) on either a cluster of 5 Pi 2s (now 3s), or a single Pi when the cluster is on a road trip, since July last year.

The only downtime has been a result of power or Internet outages at home.

Anecdotal, but I now have some 36 different Raspberry Pis at my house, and while I've had two (cheap) MicroSD card failures, I haven't had an issue with any of the Pis themselves.

As long as you use quality 2A power supplies (1A for older Pis) and quality MicroSD cards, I don't think you should worry too much about reliability, and if you have a cluster with redundancy... That's kind of the point ;)

<comment><USER11> The Pis I installed are all first version, which uses SD card. I agree that things may improve quite a lot in recent iterations.

<comment><USER13> All Raspberry Pi versions (except the compute module) use SD cards.

<comment><USER14> Raspberry Pi 2 and 3 use microsd cards.

</comment></comment></comment></comment></comment><comment><USER15> I want one of these Pi Zero Motherboards, but there has been no update by the company since January

[http://hackaday.com/2016/01/25/raspberry-pi-zero-cluster- pac...](http://hackaday.com/2016/01/25/raspberry-pi-zero-cluster-packs-a- punch/)

Like others, I have a stack of Pi systems doing file storage, DNS ad blocking, webservers, home control monitoring, XMBC, etc. A clean mount would be cool

<comment><USER16> Actually there has been a bit of movement on this board - they did some initial cost analysis and it came out way to expensive ($1000 a board!) so they are going back to the drawing board.

You can follow along here:

[https://twitter.com/9_ties](https://twitter.com/9_ties) [https://twitter.com/IdeinInc](https://twitter.com/IdeinInc)

<comment><USER15> Thanks for the update. I've poked at the twitter feed, but I missed where it says the boards would be $1000 each. (Using Google Translate sometimes doesn't go well).

Maybe when the new Compute Module comes out, someone will come up with a way to put a stack of them on the same board.

</comment></comment></comment><comment> white-flame Instead of always stacking these, with shelves and/or individual standoffs between the boards, wouldn't it be cheaper and easier to just hang them, using some longer horizontal metal rods going through all the boards' mounting holes?

<comment><USER2> In the old days you had to stack them because no 2D linear arrangement allowed access to the important ports (ethernet and whatever.

Now a days with the pi3 you can run your cluster on wifi (kills performance, but performance was never the point anyway...)

So with pi3 hardware its possible to get a length of wood 2x4 and screw the pi array to the 2x4.

Of course if you want to do stuff with the hardware other than crunch bytes you'll need the stacking anyway to access all the ports.

I'm pleased with the article. Just last week I was looking at the GeauxRobot stacking hardware. Its nice hardware, although around twice the cost.

</comment></comment><comment><USER17> I have a Pi Zero-based 8 node MPI cluster (4 Zeroes, 1 Pi2) that I set up using 8086 Consulting's Clusterhat. In total it gets about 454 Gigaflops, or roughly the speed of a Pentium 4, or maybe two Raspberry Pi 2s.

It's my 3rd beowulf, but the first that fits in my hand and runs over USB. I use it mostly for exploring fractal space and attempting to (but usually failing) to approximate pi.

Both problems are well suited to doing something in the slow lane. We know Pi to trillions of places, so there's little benefit in a fast machine to calculate it to billions. It's more a great exercise learning problem solving approaches.

So far I've managed to get a working monte-carlo based approximation across the node, but it's a terrible approach to approximating pi to anything more than a few places. I've also had some luck with implementing ramanujan-type formulae, and am still working on implementing chudnovsky in distributed forms within the bounds of the zero's RAM limitations.

On the fractal side I've only really had time to look at mandelbrot, but I'm looking forward to using it to render animations for julia and to explore henon attractors.

[1] - [http://clusterhat.com/](http://clusterhat.com/)

</comment><comment><USER18> >One big thing is to make sure atime is disabled, a massively brain dead feature inherited from 1980s Unix that writes to the disk every time you read from a file.

That's not a fair criticism of the atime feature. There are lots of use cases where it is desirable.

</comment><comment><USER19> I will be very happy if I can plug a laptop charger to a RPi and much happier if 2 RPi's can share the same laptop charger.

<comment> jon-wood It's not an off the shelf thing, but it wouldn't be that difficult to build a board which takes power from a laptop adapter and spits out the right current to hook up via the GPIO.

</comment></comment><comment><USER20> It is an Ethernet switch not a hub. The author should look into running something like a rocks cluster.

</comment><comment><USER21> Nice. Need to see more hardware posts here. Please though, improve the depth of the language.

</comment><comment><USER22> As far as computing power it's for sure useless, but tag on some kind of wiring hardness/breakout board for the GPIO pins and a redundant power supply and you might have a nice cheap platform to build a control system around.

</comment><comment><USER4> SETI@home style / GRID-computing could be used to run jobs on numerous idle Raspberry PIs (the location doesn't matter aka no cluster setup needed).

</comment><comment><USER23> This kind of stuff is what erlang lives for. Perhaps that might be something to consider ? But then again, running a couple of vm's is way better.

<comment><USER24> Check out [http://nerves-project.org](http://nerves-project.org), an Elixir framework for Raspberry Pi and BeagleBone Black.

</comment></comment><comment><USER25> Weird that none of the prices he lists are the same when I go to amazon or newegg.

<comment><USER26> Not that weird considering Amazon prices fluctuate daily (even throughout the day e.g. higher prices during lunch hours). Newegg likely does this as well.

Also note that his prices are listed _per unit_ , and the products are not always sold "per [rpi] unit". For example, a 7 port USB hub powers 7 RPi units, so unit price = (item price) / 7.

<comment><USER25> Ah, that makes sense. I was thinking they were per-unit, not per rpi. They add up better that way.

</comment></comment></comment><comment><USER27> Alternatively titled: "How to build a desk toy that functions better as a conversation piece than a practical replacement for a Hyper-V/Xen lab"

<comment><USER28> Hyper-V doesn't support ARM as far as I know. Also, do you know of a good Xen on ARM host? (honestly asking, would be cool to find one for testing cross- compiled builds)

<comment><USER27> [http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensi...](http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions#Hardware)

Probably one of those.

<comment><USER28> Sorry, I think you misunderstood? I meant hosting provider. If you have to go buy your own hardware then you may as well do precisely what the OP just did.

<comment><USER27> Scaleway like the other guy said. Yes, I misunderstood. Also the Pi is not a supported Xen host to my knowledge.</comment></comment></comment></comment></comment>


Saying 'no' to keep crap out of products - <AUTHOR> https://blog.orangecaffeine.com/the-art-of-saying-no-db012a22cd28#.sbmur7q01 <comment><USER2> Is it actually possible to know ahead of launching a feature that it'll be crap, or is it that most features get launched and the ones that are receievd well by the users are deemed "good" while the ones that aren't are considered "crap"?

We've all written things that we thought would be awesome that turned out to be disregarded by users, and deployed things we thought were pointless nonsense that turned out to be fantastically engaging features that users rave about. Can you _really_ tell before you actually get proper feedback (eg the user seeing and using the feature, not just asking if they might like it because users say yes to everything)?

<comment><USER3> Yes it is - when you're developing a feature not for the users but for advertisers or monetization purposes.

<comment><USER4> I agree. That's a big subset of "crap" features you can know are bad beforehand. After all, you know exactly, if a given feature is designed to _help_ the user, or to _hurt_ the user, or even strong-arm them for money.

</comment></comment></comment><comment><USER5> This reminds me a bit about what was mentioned in the book "good to great". A company, or in this case a product, can do everything at a level a little above garbage, or it can focus on its core competencies and be the best at something.

The more features you add to a product, (in my mind) the worse it becomes.

<comment><USER6> I think it depends on the product. There are times that I want a Swiss knife. The office suite is a good Swiss knife for me, I want it to do a bunch of stuff ok, I don't want it to be the best layout tool, or presentation builder, or editor, I just want to be able to do something ok looking fast for a number of varied cases.

There are times when I want the really great at one thing too and I don't want my Trello board to be a social network or so. I think it is more about identity of the product.

</comment></comment><comment><USER7> I've had a long career building products, and noticed a trend. When I build things that other people say that people want, they have been failures. When I build things that I want to use myself, other people tell me (almost without exception) that it will be a failure, yet they turn out to be successful.

<comment><USER8> Part of me wants to argue that this is anecdotal, though I've seen something somewhat similar in my experience. I will say this, building something you want yourself gives you a particular insight into the need that is very difficult to get when finding out what other people want.

Building personas of users is designed to help this, but you really have to be able to embody the persona.

Plus, isn't it just more fun building stuff you yourself want?

<comment><USER7> > isn't it just more fun building stuff you yourself want?

Of course!

But the interesting point here is when I propose building things that I want, my colleagues insist that nobody else will ever want it, that I'm a unique snowflake in what I want. And yet when I do, they do. It happens over and over. I can't explain it.

</comment></comment></comment><comment><USER9> The hard thing as a PM is where to draw the line of something being needed or not. In idealistic world we all want to build just features that are used by most of the users. But is that most 90%? 51%? What if a core feature is needed by just 10 or 25%?

The same goes for the kill line. Do you remove something if it's used by less than 50% of users? Or is it 10% or 1%?

Values and priorities are nice, but even with those well listed the questions above stay.

Intercom might talk about saying no, but it has grown from a simple single- feature product to an OS where mos people use just a small part of it. Preach or practice?

</comment><comment><USER10> I'd really appreciate an article that explained how to say "no" to stakeholders diplomatically. Most engineers and product people understand the danger of feature bloat but are at frequently at the mercy of people who don't.

I lead a small engineering team at a product company working for non technical cofounders and this is the hardest thing about my job. Two policies I've implemented have helped me a great deal:

1) We must thoroughly discuss the problem before proposing solutions

2) New features must be discussed and fully fleshed out at least three weeks before they can be built.

<comment><USER8> Maybe you're not saying 'no', you're saying 'not now'.

Is it possible to go to the co-founder and suggest that only features which customers will use/buy get built? If it doesn't add to sales, or have some actionable metric as to why it is being added to the product, it doesn't get done. This works for engineering too. No 'refactoring' if it doesn't have a measurable benefit to the company.

By taking this approach, the hope is that 1) the co-founder takes the time to consider what is most important, 2) can articulate that to you 3) the dev team understands why they are building the feature.

For each feature in dev, have the co-founder write out the reason it is being built.

Here's an example on the current product I'm working on.

1) We're redoing the search page because x% of users leave the site from that page without taking futher action. We aim to lower that number to x%

2) We're refactoring component x because it loads on every page and is far too slow. We'll improve response time by 600ms. This will extend page view time by x seconds and increase views by x%

3) We're going to stream our large files where we currently download the entire thing which causes customers to wait for x seconds. We'll see an x% improvement in page load, resulting in x more page visits.

You get the gist.

At first, I'd suggest the co-founder 'might' just make stuff up. But if you have something in writing that you can point back to, and then go back to him if it doesn't match up, he'll hopefully start giving it more thought.

On the flip side, it is also possible that the co-founder has good reasons to request these features but maybe hasn't communicated them well to you and the rest of the dev team.

<comment><USER10> > If it doesn't add to sales, or have some actionable metric as to why it is > being added to the product, it doesn't get done

This isn't the problem. There are justifications for new features but adding too many turns the product into a mess. I'd like them to focus more on the problem we're trying to solve and think about the product holistically.

<comment><USER8> I'm comparing this to a similar situation I was in recently trying to advise a group to focus on one problem, not all the possibilities.

That is the simple statement, as you said '[how does this feature address] the problem we're trying to solve'?

Sometimes adding too many features turns the 'product into a mess', but I would argue that is not always the case. It sounds to me like you have very little confidence in the co-founder to make the right decisions. That may be well justified, it may not be. Sorry I can't help any further.

</comment></comment></comment></comment><comment><USER11> Picture is unrelated.

A classic "no" \- Bob Lutz at GM killing a voice-input system for GM cars which let the driver operate most of the auxiliary systems by voice. It worked, but was just too clunky and embarrassing to use.

</comment><comment><USER12> I really think it is important to say 'No' and agree with the author but I think you can do this if the product you propose can be extended.

I think the framework/product should propose some configurability or some extension points. Then your product can stay pure and people can still adapt it to fit their need. If you propose an API then it can be integrate with other product/tools/developmnent. If it embark scripting ability then the user can modify it's behaviour...

<comment><USER13> My impression was don't over-extend a product, such as Evernote or LinkedIn have done, and try to keep your product focused - similar to the Unix philosophy of "Do One Thing and Do It Well".

<comment><USER14> I'm pretty sure you implied that, but just to be sure, I'd like to point one thing out: Doing one thing well won't help your product if there is no way to use other "things" that do their thing well. It seems like products often suffer from feature creep when there is no common way of interfacing with external products. I haven't used Evernote, but maybe the lack of proper ways to interface with external software on each platform was the main reason for them becoming bloated. Android with the intends system seems like one way of doing that but I feel it might not be powerful enough for most problems. Also you have no good way of guaranteeing the user experience if your surroundings might change in unpredictable ways. So it's not that easy.

</comment></comment></comment><comment><USER15> Counter point: WeChat. They keep piling new things into it just to screw around, and find gold in hilarious places.

Just build for yourselves. People are mostly like you.

</comment><comment><USER16> I've also seen zealous appliers of this principle turn their product into crap by saying no to everything.

<comment><USER17> Have any examples? Curious.</comment></comment>


IBM's reputation at risk in wake of census bungle - <AUTHOR> http://www.smh.com.au/business/ibms-reputation-at-risk-in-wake-of-census-bungle-20160812-gqqxpa.html <comment><USER2> A few years ago I worked on a project where my company was contracted to build a small piece of a much larger system by IBM. IBM was managing many different contractors. We needed a way to "securely" show a picture, aka not using the pre-installed image showing app. So IBM spent tens of thousands of dollars to build a new Android app that displayed a picture passed to it via an intent. I told our product guy that I could build it in an hour for free, but he said it wasn't our problem. In our twice-weekly status calls, we would ask about when the "image showing app" would arrive, and we heard all about the minutiae of building that app, from getting a PM, to finding a partner in India, to QA, to issues with the app, back to the contractor, and finally to us, where it was buggy as hell and never worked right. Completely mind boggling, and very enlightening about how IBM (and I'm sure other big vendors) operate.

<comment><USER3> Presumably that kind of process rigor is necessary in massive, high-scope projects, and the fallout we see is chiefly from the inability to apply a more lightweight process for tiny projects?

<comment><USER4> I'd call it process theater rather than process rigor.

<comment><USER5> I don't disagree with the description "process theater", but its a common part of the overhead of enterprise contracting (often as a result of the demands of the customer, and often central to the compensation structure), and even a bigger part of the overhead of enterprise contracting that involves subcontracting.

"Securing revenue through enterprise development contracts" is a _different_ discipline than "software development", and optimizing an organization to perform on the former can involve compromising on the latter. (The entities that understand software development well enough to align incentives of contractors well with effective delivery of value are usually the ones that don't need to contract development out in the first place.)

</comment></comment></comment></comment><comment><USER6> IBM's reputation is utterly nonsensical. I've worked for a company bought by IBM and for companies that used IBM's products/services and they've never been anything other than mediocre, sometimes outright bad, with costs that are astronomical.

My recommendation is that if you ever think about using an IBM product, hire a developer instead. You'll pay less money for a developer than you would pay for an IBM product, and after they're done implementing a the product better whatever unreliable enterprise clusterfuck IBM would shoehorn into your "solution", you have a developer who can work on other things.

<comment><USER7> Anybody who's been subjected to Lotus Notes is well-aware of the average quality of IBM software.

I found this headline kind of amazing because I was like, "reputation at risk? This has been _my_ impression of their reputation for decades."

<comment><USER8> Killnotes.exe was good ibm software though

</comment></comment></comment><comment><USER9> Wait, wait...

" _" If you think about something like what goes on in Nauru, if a child is injured it's not the government that's responsible, it's Serco, Transfield or whoever else," she said, referring to the private contractors that run Australia's offshore detention centres._"

Private offshore detention centers for children?

What's going on down there?

<comment><USER10> Been all over the news last couple weeks,

[http://www.abc.net.au/news/2016-07-25/four-corners- evidence-...](http://www.abc.net.au/news/2016-07-25/four-corners-evidence-of- kids-tear-gas-in-don-dale-prison/7656128)

[https://www.theguardian.com/australia- news/2016/jul/25/teena...](https://www.theguardian.com/australia- news/2016/jul/25/teenage-boys-were-locked-in-cells-when-sprayed-with-teargas- in-nt-juvenile-centre?CMP=soc_567)

[https://www.theguardian.com/australia- news/2016/jul/26/abu-g...](https://www.theguardian.com/australia- news/2016/jul/26/abu-ghraib-images-children-detention-australia-public- inquiry)

[https://www.theguardian.com/australia- news/2016/aug/10/the-n...](https://www.theguardian.com/australia- news/2016/aug/10/the-nauru-files-2000-leaked-reports-reveal-scale-of-abuse-of- children-in-australian-offshore-detention)

[http://www.npr.org/2016/08/11/489584342/claims-probed-of- bru...](http://www.npr.org/2016/08/11/489584342/claims-probed-of-brutal- conditions-for-refugees-on-island-of-nauru)

[https://en.wikipedia.org/wiki/Immigration_detention_in_Austr...](https://en.wikipedia.org/wiki/Immigration_detention_in_Australia)

<comment><AUTHOR> The youth detention centre issues in the Northern Territory are completely unrelated to the immigration detention centres on Nauru.

</comment></comment></comment><comment><USER11> In my mind, this kind of fuck up seems consistent with IBM's (and other mega vendor) reputation when it comes to this kind of contract.

It's more reflective of poor procurement discipline than anything else -- whomever put out the bid let lobbyists or high level people do their thing without appropriate governance or penalty. It's a relatively small contract, and without clear penalties for poor performance this type of large contractor will do as little as possible.

<comment><USER12> I'd wager there were other complexities purely from the govt side. When healthcare.gov launched and crashed, it was uncovered that the final definition of the laws and rules were still coming weeks before the launch and had changed dozens of times throughout.

From a purely technical perspective, there's no way to know if something "works" if it hasn't been defined or - worse! - is constantly being redefined on the fly.

<comment><USER13> >I'd wager there were other complexities purely from the govt side.

That is true for oh so many government IT projects. The part that I still struggle to comprehend is "Why on earth aren't the IT companies, like IBM, telling governments that what they're asking isn't actually possible?"

The Danish government have an insane amount of failed IT projects. Most of them fails do to the complexity of laws, rules and requirements. We even know that they'll fail before they even start. Most good developers are able to predict this with great accuracy. So why aren't IBM and other telling governments that what they're asking isn't possible?

My best guess: Because their lawyers will ensure that they're paid anyway, and because they have absolutely no pride in their work.

You can't hurt the reputation of IBM, or at least it won't matter, because their competitors have equally poor reputation, and no one else is going to bid on the jobs they take on.

<comment><USER14> IBM is best at writing iron clad contracts that mean precisely that -- no matter the outcome, they get paid.

<comment><USER15> IBM: You can buy better, but you can't pay more.

<comment><USER16> Incredible Billing Machine

<comment><USER15> We must have had a dozen of these in the 1980s. An internal IBM one was I've Been Moved. Others I can still remember include Install Bigger Memories, It's Better Manually, Itty Bitty Machines, and Idiotic Bumbling Morons....

</comment></comment></comment></comment></comment></comment></comment><comment><USER17> An interesting thing to note that is that Australia selected IBM to run the digital Census (which, surprise, failed horribly) whilst one of their states, Queensland, has an explicit IBM ban due to an ongoing $1.2 billion dollar contract debacle[1].

As an Australian, I hope we can move far away from such contractors. The amount of waste is insane. It reminds me strongly of the initial Healthcare.gov contracting failures.

[1]: [http://www.itnews.com.au/news/queenslands-ibm-ban-lives- on-4...](http://www.itnews.com.au/news/queenslands-ibm-ban-lives-on-420969)

<comment><USER18> I think the Healthcare.gov comparison is interesting, because to my knowledge it's the exact opposite situation.

On Healthcare.gov, there was no primary contractor who had authority to manage the other contractors building various pieces. Solution: hire an IBM.

On the Australian census in the article, it seems like they only contracted IBM to both manage and build the project.

My takeaway is that you can get a decent result in large software projects, but (1) hire a party whose sole role it is to manage / conduct ongoing audits of work, (2) set the contract such that the managing party has a mildly adversarial relationship with the implementing parties, & (3) cover off cost overruns in the initial contract.

<comment><USER17> I may be wrong but from what I read the solution for Healthcare.gov didn't involve a primary contractor, it involved changing policies to reflect start- ups[1].

The Healthcare.gov debacle and resulting "start-up culture" resulted in the US Digital Service, which terms themselves as "a startup at the White House".

[http://www.theatlantic.com/technology/archive/2015/07/the- se...](http://www.theatlantic.com/technology/archive/2015/07/the-secret- startup-saved-healthcare-gov-the-worst-website-in-america/397784/)

</comment></comment></comment><comment><USER19> The funniest thing about the census debacle is that enough people mistook _census_ for _Sensis_ (a moderately well known brand that publishes our Yellow Pages) in large enough quantities that sensis.com.au completely fell over.

<comment><USER20> Interesting. Sensis runs on AWS.

<comment><USER21> Maybe they had a limit on how much they would spend to serve pages?

</comment></comment></comment><comment><USER22> $9.6m AUS sounds like peanuts for this — something is surely wrong here.

<comment><USER23> I agree. The price seems very low to me also, especially since they can't use a cloud provider. I'm assuming they are running this in some already established data center that they own and control.

Having to be in a DC alone makes them much more vulnerable to a DDoS (I remember our provider would just cut us off first and then ask questions later is they spotted bad traffic).

<comment><USER15> It was hosted by SoftLayer, which IBM bought in 2013.

[https://en.wikipedia.org/wiki/SoftLayer](https://en.wikipedia.org/wiki/SoftLayer)

</comment></comment></comment><comment><USER24> I started working at IBM ~3 months ago, as a DevOps Engineer. Joining IBM was probably the worst mistake that I have ever made. My group (IBM is a big company) is a complete cluster-f. No leadership. Managers have no knowledge of the technology that they are in charge of. Worst of all, we are forced to use IBM's proprietary, shitty tools (Notes, SameTime, Rational Team Concert, UrbanCode Deploy, IBM-specific JRE, Retain, ect, ect.). I started looking for a new job after two weeks. I would not buy anything from us.

</comment><comment><USER25> If anything it's just cementing their reputation as a complete joke.

Corp sector in Australia is on the tail end ridding themselves of IBM nonsense after a decade of utter incompetence.

Unfortunately the gov sector catches on slowly.

Sadly a lot of places don't learn though and just replace IBM with TCS/TechM which is somehow even worse.

<comment><USER26> I've dealt with a couple of customers of ours in Australia that have used IBM services for their IT. It's been a complete disaster. I don't know whether they are all that bad, or Australia gets the C team, but, wow.

And don't get me started on TCS. Fucking shitshow.

</comment></comment><comment><USER27> Where is Watson when you really need it. I would be much more impressed by "Watson successfully saves an IT project screwed up by IBM management" than an article about how Watson diagnoses cancer.

<comment><USER28> I guess if Watson was really intelligent it would have not picked IBM in the first place, leading to a paradox

</comment></comment><comment><USER29> Not long ago there was a thread about IBM PhotoStore system from 67, a Terabit of storage on photographic film: [http://www.computer- history.info/Page4.dir/pages/Photostore....](http://www.computer- history.info/Page4.dir/pages/Photostore.dir/index.html)

I miss this IBM.

</comment><comment><USER30> "The detail of exactly what happened has not yet been independently reviewed. And as we saw in other stuff-ups, such as Queensland Health, when it was independently reviewed the blame was shared around quite a lot," Mr Noonan said. The attack was either a foreign or locally planned denial of service, or just a "large load" that appeared to be a denial of service but was in fact people trying to fill out the Census, he said. It "beggars belief" IBM's data centre could not handle a denial of service attack, he added."

There's been a bunch of discussion in Australia that there likely had been no "attack" and that it was simply just terrible planning around expected loads the night of the census.

</comment><comment><USER31> This must be all the more galling for IBM as they literally have more than a century of experience with census jobs: [https://en.wikipedia.org/wiki/Herman_Hollerith](https://en.wikipedia.org/wiki/Herman_Hollerith)

</comment><comment><USER32> haha, all they do is problems with outsourcing.

This happened in canada. [https://www.google.ca/webhp?sourceid=chrome- instant&ion=1&es...](https://www.google.ca/webhp?sourceid=chrome- instant&ion=1&espv=2&ie=UTF-8#q=canada%20phoniex)

</comment><comment><USER33> IBM also caused $100,000,000 waste on New Zealand's failed INCIS police computer system.

</comment><comment><USER8> they've been rearranging the titanic deck chairs for quite some time now

</comment><comment><USER26> Maybe somebody will finally get fired for buying IBM

</comment><comment><USER34> No one ever got fired for buying IBM!

Too soon?</comment>


Linux Perf_events Off-CPU Time Flame Graph - <AUTHOR> http://www.brendangregg.com/blog/2015-02-26/linux-perf-off-cpu-flame-graph.html <comment><USER2> As a general observation, and coming from a background spending a lot of time on performance troubleshooting, Brendan spending time doing performance work on Linux is a huge win for Linux bods everywhere, and a big loss to the Solaris community.

</comment><comment><USER3> Can anyone who has dealt with Linux perf _and_ Windows perf comment? I've spent a fair amount of time dealing with xperf/WPA and spelunking ETLs and find that it's a really nice environment. Does something similar exist for the Linux world?

This seems similar to wait chain analysis, although it's something that I'm still trying to get better at, so I can't really draw any parallels here.

<comment><USER4> I've done some Xperf/ETW, and saw many similarities to Linux perf: various events to trace, in groups, and a trace->dump->post-process workflow.

Linux perf has good support for profiling performance monitoring counters, from instructions per cycle to uops events. I haven't seen them instrumented on Windows, other than by Intel VTune. Linux perf can also do kernel & user- level dynamic tracing; I've heard such support exists on Windows, but not seen it done yet.

The event documentation with Windows ETW looked better. On Linux, some of the events (tracepoints) have no documentation other than the kernel source. (But, at least you have kernel source. :)

Symbols is also often more difficult to get going on Linux. Bruce Dawson (who can probably answer this Q better than anyone) has written about it in the past: [https://randomascii.wordpress.com/2013/02/20/symbols-on- linu...](https://randomascii.wordpress.com/2013/02/20/symbols-on-linux-part- three-linux-versus-windows/)

perf, and Xperf, are pretty good, but what I think they really need is more kernel level programability, to do (more) custom in-kernel filtering and summaries. DTrace has shown the value that this can bring. It should lower the overhead involved, and allow these tools to be used more often. Linux perf may get there with eBPF (although it does currently have in-kernel counts and custom filters)...

<comment><USER5> You called?

I didn't know you did Windows profiling.

I do some profiling on Linux, although nowhere near as much as on Windows. My observations of the differences are:

I love how easy it is to measure CPU performance counters on Linux, whereas this requires VTune or similar on Windows and is generally painful. I've heard that ETW does support these performance counters but that Microsoft has yet to expose them, which is sad.

I love how easy it is to get counters like context-switch counts on Linux. Just run your command under perf and the results are available the instant it finishes.

perf top is also a wonderful thing for real-time investigations.

So, Linux windows for quick access, access to real-time information, and CPU performance counters.

xperf/ETW/WPA wins a lot of points for the UI. This allows exploring details. For instance, on Linux I can trivially see how many context switches there are, but zooming in on an individual context switch to find out when and why it happened... I wouldn't know how to begin.

Brendan said: > The event documentation with Windows ETW looked better.

I thought that was curious because I found the documentation of the WPA columns so atrociously non-existent that I wrote a series of blog posts to summarize what I had figured out: [https://randomascii.wordpress.com/2012/05/11/the-lost- xperf-...](https://randomascii.wordpress.com/2012/05/11/the-lost-xperf- documentationcpu-scheduling/)

At least they have tooltips now -- that's progress.

So, Windows needs access to CPU performance counters, kernel level programmability, and ability to quickly get back statistics. Linux needs a UI for exploring the data from "perf record", and Linux needs a better story for retrieving related symbols and source automatically.

In short, both platforms should try to copy the best features of the other because they are both missing vital items.

BTW, nice job on the idle-time flame graphs, but I'd definitely still give xperf/ETW/WPA the advantage, especially when tracing wait-chains that span processes.

<comment><USER3> Thanks Bruce/Brendan. Interesting info to know.

Another general perf question: How do you generally explore perf problems? I find that I can narrow down performance differences to the point of "we spent more time in function foo" fairly easily. Understanding _why_ can be difficult (was it an intentional change? Should we have been in there at all? Did some data structure change this time around?).

All in all, I feel like a detective, and don't have a rigorous method to apply when looking at problems. Just wondering if anybody feels similarly or if with more knowledge and understanding will come clarity.

<comment><USER5> I've got no silver bullet for that. Recently I found a bunch of functions that were running slower on Windows. I eventually proved that their code had not changed and they were being run the same number of times as before. I had to use perf on Linux to prove that the i-cache miss rate was elevated due to changes elsewhere, making the identical code run slower.

Having a good sense of how long something "should" take is certainly important, but investigating small performance regressions is always hard.

That said, there are often a lot of big performance problems, and those are easier to find and often easier to understand, and more important to fix. Hey, maybe we shouldn't have a 50 MB global variable in each server instance -- whaddya think?

Being a detective is good. Apply the scientific method, form hypotheses, test and reject them, rinse repeat. That's certainly way better than the off- applied "random guess based on Internet search" technique.

Brendan's book is a good read.</comment></comment></comment></comment></comment>


Comparing Haskell and Node concurrency performance - <AUTHOR> https://www.fpcomplete.com/blog/2016/12/concurrency-and-node <comment><USER2> > Which leads to the 'callback hell' that we all know and hate. Part of the > bill-of-goods we accept when using Node is that in exchange better time and > space characteristics, we lose the thread as an abstraction.

Perhaps my head has been stuck in javascript/node land for too long but I think accusations about javascript producing callback hell now seem a bit disingenuous even for relative novices to the language.

It's 2016 and there are many well documented and widely adopted solutions arising from external libraries and developments in ECMAScript. Thanks to transpilers like Babel/Typescript we can even shoehorn these new ECMAScript features into older browsers.

<comment><USER3> I never had callback hell even before modern Javascript times. I simply used named functions instead of inlining everything. Nesting level: 1, maximum 2 if I felt it was okay. And modules, modularization is key or it gets too complex.

So the _lexical structure_ of my code was linear - while the _runtime structure_ was nested at arbitrary levels. There never was a reason to represent the nested runtime structure in the written code.

I also didn't attempt to use node.js for things it wasn't made for, like compute-intensive tasks or implementing business logic. The good old chat server for ten thousand people was an often used example for node.js programming for a reason - lots of I/O, little processing.

 _Note: I don 't write code like that any more in ES 2015. I also don't use classes, prototype, this, bind, apply - only functions and (lexical) scope (with an eye on capturing only as much scope as I need). Which is the opposite of the above described method where lexical scope was not usable, but with the methods available now the code still is "flat", so that's why I switched._

<comment><USER4> That sounds like its own form of hell to me. Ideally, a lexical structure helps visualize and understand the runtime structure. Anything that obfuscates that is a recipe for disaster.

<comment><USER3> Runtime structure can be arbitrarily nested, how do you want to show that in code structure? That makes no sense. You presume a static structure of who calls whom. It also isn't very flexible (refactoring, implementing change requests).

The key was of course to come up with great modularization, of course you would not want to do that with "flat code", the complexity of what function is where would be (or would have been, since I no longer need to write in that style) overwhelming.

<comment><USER4> I think that is part of the point. You want to do things that make it obvious when the runtime structure has gotten arbitrarily nested. Closure callbacks actually help there, since they make it someone visible and easy to "smell."

That is, if you have the same nesting at runtime, but it is just somewhat obscured by the naming style that you did, that sounds problematic to me. Ideally, you find structural ways to get rid of that nesting. (I said elsewhere that I'm a huge fan of first class queues. There are other options. Callbacks are one. And realistically, what you describe is an option, too. None of them are intrinsically bad.)

</comment></comment></comment></comment></comment><comment><USER5> It feels like this generation has never had to use Windows 3.1 Or maintain an event loop application long term.

There are good reasons we went to thread based models - developer productivity and safety. Event loops are fine for toy demo's, or very carefully managed products (trading systems, NGINX) but not for use as general purpose hammers.

Every single bit of extra friction and cognitive overhead costs you dearly a few years down the line. We scrambled away from this stuff as soon as we could, and there's no good reason to go back.

<comment><USER6> Having done programming with both threaded and event-looped systems, I think that event loops (when done well) cause less cognitive overhead. With threaded systems, I was constantly worrying about how things need to be locked and what happens if two things run simultaneously. Event-looped systems make the break points explicit, so I _know_ precisely when other things might run.

"Callback hell" is IMO a terrible way of doing event-loop systems, as the nesting can get confusing. For implicit event loops like Node, I strongly prefer the Promises approach (preferably with async/await sugar); for explicit event loops (I only have practical experience with Arduino, though I've also a passing acquaintance with the classic 69k Mac as well) I like to build a set of event-driven state machines with cooperative multitasking. If properly designed, these keep everything nicely separated so you can follow the logic without any trouble, while also avoiding the concurrency concerns of a threaded system.

<comment><USER4> I don't actually see how promises help. In particular, it seems to encourage people to build systems that basically horde promises in odd places.

I greatly prefer first class models for where things queue up, and then to use producer/consumer objects against those queues.

<comment><USER6> I'm not sure what you mean by "horde promises in odd places"\--you use a promise whenever something is happening asynchronously, certainly, but I'm not sure what's odd about that. Maybe this is an antipattern I haven't encountered yet?

I think producer/consumer and promises cover two different use cases (with some overlap). Indeed, when I have a work queue it frequently involves promises: queueing a work item returns a promise which will resolve when the work is complete, and part of executing a work item is returning a promise so the consumer knows when the task is done and can start in on another one.

<comment><USER4> I have seen paths that will generate upwards of 10 promises and then put them together at the end. In theory, this shouldn't be a problem and the code was readable enough. Reasoning about all of the different ways backpressure can happen was not as easy.

Compared to knowing that you have a set of queues that ultimately feed into other queues. It is much clearer to reason about the throughput of individual queues and take that into consideration when designing new queues in the system.

It is basically like someone throwing a ton of outlets on a wire going through a room. I mean, yes. You can do that. Often won't even cause issues. However, for large enough systems, you ultimately need to know what the load on that circuit will be and it is not acceptable to put yet another plug extender there.

</comment></comment></comment></comment></comment><comment><USER7> I don't think this comparison is really fair at all... it seems to be cherry picked to point out what is already known to be a bad use case for node. In terms of the multiple async calls, Promises and async functions takes care of that, not to be confused with the reference to the `async` library.

First, using clustering and memoization would improve the throughput a lot. I did something similar when adapting a JS based script library to be used in node, because I knew it would lock the main loop otherwise. Beyond this, cpu intensive work should be avoided in your service loop regardless. It's best distributed to an RPC/Worker pool.

In terms of scale, node scales as well or better than a lot of frameworks, it's only that you will usually want to use similar techniques locally as well as remote.

Another poor example is when you need millions of references in a single thread, Node will die spectacularly. That doesn't mean it shouldn't be used for many use cases, it only means that it's bad at some of them.

I find that node is _great_ as an intermediate/translation layer... your UI talks directly to node, tightly coupled.. then node can translate against backend databases or other services as a gatekeeper for your front end. It allows you to make the data the shape that is most convenient, with the least amount of disconnect of thought and approach.

It's also pretty great for certain types of orchestration control and even in the proof of concept stages of applications. Doing a first version of almost anything I've tried in Node is usually much faster than alternative platforms. And often performs well enough to stick with it. Developer productivity is more important than absolute scale at the beginning, and if you have a plan to scale horizontally, you can do that for a while before you need to break off other optimizations.

<comment><USER8> _Doing a first version of almost anything I 've tried in Node is usually much faster than alternative platforms._

Do you say this because of the libraries available for it? (curious, wanting to jump to node).

<comment><USER7> I mean faster to get up and running... development time that is. Mostly, for me, because I'm doing front end work in JS, using mostly the same tooling npm, babel (though with webpack on front end now).

It's just so much faster to get going if you're developing the full stack, and already in JS heavy land anyway. Not having to context switch for the backend is huge... being able to use a document/object/json database isn't as big of a boost but still nice. On the db side, I've been using the template wrappers so that I can write a simple query and it turns it into a parameterized query returning a promise.

        async function getRecords(baz) {
          return await sql.query`
            SELECT
              a,
              b
            FROM
              foo
            WHERE
              foo.bar = ${baz}
          `;
        }

So, I still have to think about some SQL, but still usually better than trying to twist ORMs into shape.

Overall, for the past 6 years or so (since 0.8) I've been using Node pretty heavily (moving from more C# on the backend) and really haven't missed it at all. Even though the core .net and more open-source stuff has been interesting... Managed to dockerize a few trivial .Net apps using the dotnet onbuild base containers.

If you're using windows, the only gotchas are you need a C++ build environment (Visual C++ 2015 Build Tools, checking all options) and Python 2.7.x in order to build any binary modules... most of which now run without issue on windows, was a _much_ bigger problem in 0.8-0.10 ...

</comment></comment></comment><comment><USER9> This article is misleading. Here's the real problem: [https://github.com/AndrewRademacher/fpco-article- examples/bl...](https://github.com/AndrewRademacher/fpco-article- examples/blob/master/concurrency-and-node/node-async/src/starvation- main.js#L22L32)

Anyone who understands JavaScript can see that the recursion invoked in the slow route is not asynchronous (each recursive invocation keeps piling onto the call stack without ever releasing it)! You'd have to use process.nextTick (or setTimeout) if you wanted to recurse asynchronously without spawning a new process...

For these kinds of unusual, heavy computations, though, you'd be better off using the child_process module to spawn a new process and do the recursion inside that process so that it doesn't block the main event loop.

This has nothing to do with starvation. Node.js just has a completely different approach to this kind of problem.

<comment><USER10> > Maybe Haskell does some sort of automatic multi-threading behind the scenes > - If so, I'm not sure that's a good thing

GHC (the leading Haskell compiler) provides an extremely advanced green threading system with the runtime. [http://haskell.cs.yale.edu/wp- content/uploads/2013/08/hask03...](http://haskell.cs.yale.edu/wp- content/uploads/2013/08/hask035-voellmy.pdf)

> I like the way it's done in Node.js - Explicitly.

If you wanted to do it this way in Haskell, there are a number of monads that make it way more convenient than doing it in Node. Cont in particular can be used for cooperative concurrency. No one really uses these outside of niche cases, however, because threads are almost always a better abstraction.

<comment><USER9> Maybe, Golang also has similar advanced multi-threading constructs.

But I do a lot of REST API (and WebSocket) work and all of the workload that happens inside my Node.js program is extremely lightweight.

If I need to perform some heavy computation, I will offload it to a separate child process - Node.js forces me to put that code in a separate file/module but I actually like this because it encourages separation of concerns. It feels very natural so I don't really need any other special constructs.

I looked into goroutines a while ago; it looks cool, but I probably wouldn't use them much because I don't like the idea of having code from the same source file splitting off into multiple processes/threads; it makes is harder to read and reason about the code (this is a bit like what happens with multi- threaded code when you have mutexes all over the place).

To me, this feature has the same utility value as the ability to define multiple classes per source file - Ok, that's cool, but is it a good idea to do that?

<comment><USER10> I don't really have strong feelings about source file organization, but the way it works in Haskell is that all IO actions have type "IO a", where "a" is the type of the action's result. To do multithreading, you just do e.g.

        foo :: IO () -- optional type declaration
        foo = forever (print "hi")
        forkIO foo

Now Foo will run in its own green thread forever. Very simple. This works with any IO action. You can put it in whatever file you like. For communicating across threads you have many great options like STM. If you like Go-style chans, there are libraries that provide various types of chans.

</comment></comment></comment></comment><comment><AUTHOR> Previous comments on threaded vs event-loop concurrency abstractions are relevant:

[https://hn.algolia.com/?query=why%20events%20are%20a%20bad%2...](https://hn.algolia.com/?query=why%20events%20are%20a%20bad%20idea&sort=byPopularity&prefix=false&page=0&dateRange=all&type=story)

[https://hn.algolia.com/?query=why%20threads%20are%20a%20bad%...](https://hn.algolia.com/?query=why%20threads%20are%20a%20bad%20idea&sort=byPopularity&prefix=false&page=0&dateRange=all&type=story)

[https://hn.algolia.com/?query=threads%20vs%20events&sort=byP...](https://hn.algolia.com/?query=threads%20vs%20events&sort=byPopularity&prefix=false&page=0&dateRange=all&type=story)

</comment><comment><USER11> On one thread, I am defending FP from critics who are unaware from the state of the art.

EDIT: Just to make it clear early on, I agree with the article's conclusion that Nodejs is not as good at compute heavy workloads as Haskell. I simply object to any use of "the nested callback problem" as valid in 2016. It's an issue exclusively for legacy code and developers who take pride in writing outdated code.

It seems only fair, then that I also should defend Javascript from people obviously unaware of the state of the art in pseudo-imperative programming. And by state of the art, I mean "has been around in some languages for 3+ years."

The example:

        request('http://example.com/random-number', function(error, response1, body) {
          request('http://example.com/random-number', function(error, response2, body) {
            request('http://example.com/random-number', function(error, response3, body) {
                ...
            });
          });
        });

But modern Javascript (before you start, yes, it runs on every browser with preprocessing, which is normal for this ecosystem) would make it look more like this:

        // rp is a request promise, multiple options for creating them
        async function make3StaticRequests() {
            try {
                var res1 = await rp('http://example.com/random-number')
                var res2 = await rp('http://example.com/random-number')
                var res3 = await rp('http://example.com/random-number')
                // ...
            }
            catch(error) {
                // ...
            }
        }

        // And of course the promise library allows for many things
        // you'd like with applicative functors, like binding groups
        // of operations together and evaluating them all.

        function randomNumberPromise() {
            rp('http://example.com/random-number')
        }

        async function make3StaticRequests() {
            var [res1, res2, res3] = Promise.all([randomNumberPromise(),
                                                  randomNumberPromise(),
                                                  randomNumberPromise()])
            // ...
        }

I don't really understand why people feel comfortable writing up comparison articles without doing sufficient research into what they're comparing things to.

That said, the articles point about large compute workloads starving other operations is very much true and a good example of what the weakness of V8 as a server-side programming environment brings.

<comment><USER12> For starvation there is also clustering, which can shift the load towards worker processes that aren't busy with cpu-intensive tasks, as well as simple modules such as `[https://www.npmjs.com/package/process- pool`](https://www.npmjs.com/package/process-pool`) which can be used to offload (known) expensive tasks.

While Haskell is truly better at concurrency (no need to serialise when passing messages, green threads yield not only at IO but also at memory allocations), that part of the comparison isn't very good. Spawning a cluster of NUMCORES threads using the built in cluster module would be an improvement.

<comment><USER11> Personally I think this is a total non-answer to the question. It enormously complicates the concurrency and data sharing story for a solution that literally every programming language has access to.

Nodejs has a poor story for compute-intensive loads. People need to be comfortable saying that, because it's reality.

<comment><USER12> First of all, there is no enormous complication. Clustering in node is super- easy, and file descriptors of requests are sent automatically to one of the processes in the process pool. This benchmark should have at least done that.

Secondly, the service presented there doesn't even need to take advantage of shared memory concurrency at all. This is the case for a vast majority of web service problems too: they either talk a lot to each other or do a bunch of cpu-intensive work, but rarely both.

Finally, when you use shared memory concurrency/parallelism to solve web service problems, there is a risk that the resources of a single machine will not be enough. And then you are back to serialising things and sending them through an even slower channel.

Haskell also has poor facilities for compute-intensive stuff, although they are different facilities. For example, laziness makes reasoning about performance more difficult. Space leaks are fairly easy to create unless you know the common gotchas. Most naive/idiomatic Haskell code performs several orders of magnitude worse than whats possible with optimized code. Etc etc.

<comment><USER11> > But those are not web-server problems.

I disagree. Totally and utterly. API servers are webservers, and in fact a pretty big subgroup of them. API servers can often run into computational requirements in mid request, and it sucks that ONE slow request can cause latency ripples across your entire process.

> Haskell also has poor facilities for compute-intensive stuff, although they > are different facilities. For example, laziness makes reasoning about > performance more difficult.

This is changing the subject, and ultimately a non-sequitur. "This also has problems which are different" is not actually an answer to the criticism that nodejs is bad at these workloads.

Say it with me. It's okay to say. "Nodejs is bad at compute-intensive workloads."

<comment><USER12> > I disagree. Totally and utterly. API servers are webservers, and in fact a > pretty big subgroup of them. API servers can often run into computational > requirements in mid request, and it sucks that ONE slow request can cause > latency ripples across your entire process.

Or it doesn't cause ripples, since the cluster master doesn't send requests to it, and instead redirects them to the other processes.

That, plus a process pool for known compute-intensive stuff is often good enough.

<comment><USER11> > Or it doesn't cause ripples, since the cluster master doesn't send requests > to it, and instead redirects them to the other processes.

A methodology that doesn't scale well across boxes, so only buys you so much. Demo code might benefit from this approach, but production code will do something totally different to give an API or server actual durability and uptime.

> That, plus a process pool for known compute-intensive stuff is often good > enough.

To make it "good enough" for production workloads for anything less than a trickle of traffic requires an entirely different and probably queue-based architecture. While this is often in good style, it is (I restate) _a tool available to every language environment_. If everyone has this technique, you cannot say that Node is given a pass because "if you just totally re-architect this code" then it's okay.

Nodejs's scheduler and execution model fundamentally make it worse at compute- heavy workloads. THIS IS AN ACCEPTABLE TRADEOFF. But denying it exists only misleads engineers and leads you to bad decisions.

<comment><USER12> > A methodology that doesn't scale well across boxes, so only buys you so > much. Demo code might benefit from this approach, but production code will > do something totally different to give an API or server actual durability > and uptime.

Yes, you would have a load balancer in front of several instances running on several machines. Same as Haskell.

> Nodejs's scheduler and execution model fundamentally make it worse at > compute-heavy workloads. THIS IS AN ACCEPTABLE TRADEOFF. But denying it > exists only misleads engineers and leads you to bad decisions.

I agree, in principle. But one, its not nearly as bad as this article makes it. And additionally, the given example is not convincing or very representative.

<comment><USER13> It absolutely is as bad as 'KirinDave says it is, although the article does a bad job of showing it. Node shines at IO-bound applications, sure, but let's say you want to do one big computation on 16 cores all at once.

With Node, you'd have to serialize data and pass it between child processes. And that really, really sucks. Haskell's parallelization story extends way past the "embarrassingly parallel" request handling.

A good data point based on a less contrived example is how handily Haskell web frameworks _demolish_ Node.js at JSON serialization... and not much else ([https://www.techempower.com/benchmarks/#section=data-r13&hw=...](https://www.techempower.com/benchmarks/#section=data-r13&hw=cl&test=json&l=4fthvj)).

Finally, if you're looking to do something truly and extremely CPU-bound, I'd tell you to write it in a C derivative and bind it to Node or Haskell, regardless of what your favorite language is. Optimize for speed in some places, and programmer happiness in others. A one-size-fits-all approach isn't usually appropriate.

<comment><USER12> I just spawned a least-latency load balancer in front of a cluster of N processes (one per core) for the benchmark.

[https://github.com/spion/fpco-article- examples](https://github.com/spion/fpco-article-examples)

Now it gives the same results as the Haskell solution. (Which only means this concrete benchmark doesn't reflect real world performance)

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER14> The first part about comparing the callbacks in Javascript vs do notation in Haskell is super misleading. The do notation desguars to code that looks essentially identical to the Javascript version. Sugar is not a negligible consideration in system choice, but that kinda thing just bugs me.

<comment><USER15> It desugars that way beacause of the nested structure of monadic sequencing, not because of threads or IO. We can distill the real argument down to "JavaScript must use nested callbacks for async. Haskell doesn't."

</comment></comment><comment><USER16> A lot of the readability problems with NodeJS the author mentions at the beginning of the article have been solved in numerous ways. Promise based work flows for instance allow one to define a step by step flow very similar to the counter examples provided. Packages like babel can expose things like yield and async/await which get us even closer. I'm not saying its ideal but it certainly mitigates the worst parts of the 'callback hell' problem pointed out.

</comment><comment><USER17> "Many web servers, for example achieve concurrency by creating a new thread for every connection. In most platforms, this comes at a substantial cost. The default stack size in Java 512KB, which means that if you have 1000 concurrent connections, your program will consume half a gigabyte of memory just for stack space. "

As a WebLogic developer we fixed this in the late 90s but the Volano chat benchmark was still run for no apparent reason. Somehow everyone else didn't get the message until Netty was released and people started using it.

Obviously what you want is multi-threaded execution with asynchronous I/O. Using node on multi-core systems just doesn't make a lot of sense as you end up having to duplicate your entire program on each core to get the full performance of the machine. Not unlike 512k/thread but much worse — especially if you cache anything locally in the process like template compilation, etc.

<comment><USER18> If you're going to just duplicate your process across all the cores then yes, local caching becomes an issue. Which was solved with Redis. Back in 2009.

Don't get me wrong, the fact that Node doesn't have a more efficient way of handling overhead on multicore machines is a drawback. But you pay for your abstractions. Nobody is going to argue that Node is a phenomenal solution for lightweight, multi-threaded execution. But most companies using Node seem to accept this and are fine with not utilizing 100% of their resources (hell, most don't even seem to know Node can spawn processes, in my experience).

If they care they use a framework that meets their needs.

</comment></comment><comment><USER19> > Node ... popularized the event-loop

Tcl was doing event loops quite successfully in the late 90ies and was fairly popular, back in the day.

These days, I'd use Erlang (Elixir).

This is kind of ranty, but also makes me laugh: [https://www.youtube.com/watch?v=bzkRVzciAZg](https://www.youtube.com/watch?v=bzkRVzciAZg)

<comment><USER20> This is exactly what the word popularized means. It doesn't mean invented, it just means exposed it to a large number of people who hadn't previously encountered it.

I don't know if you can claim definitively that Node introduced more people to event loops than any other technology, but it certainly is one major popularizer of the idea, and the one that's had the most impact in the past decade.

<comment><USER19> So basically people who knew nothing of what had come before, despite it being a fairly widespread technology. I think that's kind of proving the video's point.

<comment><USER20> People are constantly learning how to program, and they pick up ideas in different orders from different sources at different times in the history of a discipline. Every idea anyone has ever known was something they didn't know until they were introduced to it.

The fact that you knew it first doesn't give you a reason to be snide.

</comment></comment></comment></comment><comment><USER21> node is fast when the heaviest work is delegated to libuv or native modules (that are performant that is). If you require to do heavy work on v8, it slows down significantly.

What would I call "heavy work"? e.g: compression, serialization, encryption, image processing... tasks that are bound by CPU and not only I/O. Usually you want to delegate that to a native module and not do that yourself in JavaScript. If you absolutely have to do it in JavaScript, then you need to make sure the task is not blocking the event loop. In order to play nicer with the event loop you inject something like setImmediate or process.nextTick after certain amount of time or iterations... otherwise you will starve other tasks in the loop, notably, I/O.

node is also not a really good idea if you need a lot of interprocess communication.

It is a very viable alternative, though.

</comment><comment><USER12> Here we go, a node version that uses a least-latency load balancer and doesn't exhibit the problem:

[https://github.com/spion/fpco-article- examples](https://github.com/spion/fpco-article-examples)

Its an interesting benchmark, but it needs more work to give a more accurate picture. It would be nice if it:

      * used wrk to measure requests and latency percentiles
      * percentage of "slow" requests was tweakable
      * number of workers per core was tweakable

Then we could generate a nice chart that shows latency percentiles as fn of % of slow requests and workers per core, and compare with Haskell.

</comment><comment><USER22> Another nitpick - most Java based web/app servers have used thread pooling (or similar approaches) for at least 10 years. Seems overly simplified to say these environments always "spawn a new thread".

<comment><USER20> That saves you from thread spawning costs, but not thread overheads.

</comment></comment><comment><USER23> In regards of concurrency, I's suggest to have a look at Elixir. It's growing like weed and is offering the best programming experience you can find. Not kidding. Just try it.

</comment><comment><USER24> I guess if we're nitpicking, then here's another one:

> Looking near the top of the output, we see that Haskell's run-time system > was able to create 100,000 threads while only using 165 megabytes of memory. > We are roughly consuming 1.65 kilobytes per thread.

Those are not the same kind of threads that the author is talking about in the beginning of the article. Those a green threads and as such are multiplexed to much smaller amount of real system threads to do work in parallel. What that means is that they, for example, can't all make a system call at the same time. Go has the same issue.

<comment><USER10> > What that means is that they, for example, can't all make a system call at > the same time.

Yes, they can. The Haskell IO manager knows how to handle this. [http://haskell.cs.yale.edu/wp- content/uploads/2013/08/hask03...](http://haskell.cs.yale.edu/wp- content/uploads/2013/08/hask035-voellmy.pdf)

<comment><USER24> That's true with a caveat. It knows how to handle IO syscalls on systems where something like "epoll" or "kqueue" is available which, to be fair, is majority of what a typical server does. In the general case waiting for a syscall to return will block the system thread and no other green thread is going to be able to run on it.

<comment><USER25> The GHC Haskell runtime supports preemtive scheduling of cooperative threads ("green threads"). In non-preemtive cooperative threading, threads yield to each other. However, when a thread goes into a syscall, it no longer has the control to yield. The only way to wake up from a syscall (and thus to decide wether another thread should be scheduled, inside the runtime, and thus to get preemtive scheduling), is to send a signal to the process that's blocked in the syscall; then the syscall gets interrupted with EINTR and the runtime can do its scheduling decision, and then resume the syscall if needed.

This signal sending is done by setting up a periodic "timer signal" that sends SIGALRM to the process every 10 ms (by default).

[http://man7.org/linux/man- pages/man2/timer_create.2.html](http://man7.org/linux/man- pages/man2/timer_create.2.html)

<comment><USER24> Thanks, I did not know this.

</comment></comment></comment></comment></comment><comment><USER26> lamda, anonymous functions are so popular in JS that it migt be a suprise that you can actually name your functions. Heck you can even use them like any object, pass them along, store them in lists, return a funtion from a fuction etc.

</comment><comment><USER27> > However, Haskell doesn't impose an additional burden on the design of your > software to accomplish that goal.

 _giggles uncontrollably_

I like Haskell, but saying that it doesn't impose a design burden is incredibly misleading.

</comment><comment><USER28> is the author really just comparing single-core to multi-core? I didn't read any specs on the machine that ran the benchmarks, but assuming its multi-core, are the Haskell tests using all the cores, while node is only using 1?

node is probably not the best choice for truly CPU bound operations, but you can sometimes get by using the native cluster module to spread work over multiple cores.

<comment><USER7> For a few things, I've spun off another process, and use a pool in order to constrain the number of processes actually running at once... did this to adapt a JS based scrypt module when I was running node in windows, and non of the "native" versions of scrypt in npm would even build correctly in windows (early 0.8 timeframe iirc).

</comment></comment><comment><USER29> Node has concurrency? :P

<comment><USER30> i think you are confusing concurrency with parallelism.

</comment></comment><comment><USER9> >> The difference here is stark because in Node.JS's execution model, the moment it receives a request on the slow route, it must fully complete the computation for that route before it can begin on a new request.

With this statement, the author acknowledges that the Node.js code in the slow route was not asynchronous. The test is therefore invalid; it's comparing apples and oranges.

Node.js is more than capable of handling different requests asynchronously (regardless of whether they are fast or slow); if you have any kind of blocking or waiting around happening; then you're doing it wrong.

I'm so tired of all the anti-Node.js propaganda; it's hurting people. If I walk into one more company where some zombie tells me that they're migrating away from Node.js because "the Node.js event loop starves the CPU", I'm going to have a stroke.

In reality all Node.js 'starvation' problems can be solved with the 'cluster' module or the 'child_process' module.

Since we've been talking a lot about 'Fake news' on Facebook recently. Maybe we should start talking about how fake news is affecting Hacker News. This anti-Node.js strain is particularly virulent.

</comment><comment><USER31> Comparing apples to oranges

<comment><USER32> How so?

<comment><USER33> Node has no built-in concurrency just callbacks. The whole point is nonblocking io in node not multithreading.

<comment><USER7> Node does use threads for async io, it's just abstracted away from the main loop. The bigger issue, is the cpu bound code is a bad use case for Node, and it's known to be. There are options to run this type of code out of the main process though.

You can scale node, via the same techniques you use to scale anything across servers, you just do it sooner with node in order to better utilize a larger server, or use multiple servers sooner. Node is great for just about any io bound workflow.

</comment></comment></comment></comment><comment><USER34> So someone compared performance and programming model, and lo and behold, found Haskell to be superior.

What might be more interesting:

\- what is the salary difference between a Node dev vs. a Haskeller?

\- is there a productivity difference? does the salary over- or under- compensate?

\- is correctness a core business concern?

\- if I need to hire 10 devs, can I do that?

</comment><comment><USER35> I'm getting sick of seeing this. Node wasn't intended for compute heavy workloads...ever...at any point...for any reason. This is like the 50th time someone has decided it was appropriate to point it out by generating Fibonacci numbers (among other things).

Go watch the node.js presentation Ryan Dahl gave at JsConf 2009, he addresses this during that speech.

His opinion on the "right way" to do concurrency is a little polarizing, but, quote: "the right way to do concurrency is to use a single thread and have an event loop. this requires that what you 'do' outside of IO waits not take very long".</comment>


Families will no longer be separated at border. But where are my clients' kids? - <AUTHOR> https://www.washingtonpost.com/outlook/families-will-no-longer-be-separated-at-the-border-but-where-are-my-clients-kids/2018/06/20/9d9e59a0-74b6-11e8-805c-4b67019fcfe4_story.html <comment><AUTHOR> You can read it in private/incognito mode.

<comment><USER2> Using Brave and Chrome on mobile, I cannot.

</comment></comment><comment><USER3> I left reddit cuz of the constant political trolls...</comment>


Google to buy Postini for $625 million (100,000 businesses using Google Apps to date) - <AUTHOR> http://www.reuters.com/article/innovationNews/idUSWEN920520070709

<comment><USER2> Another boring, unsexy app going for nice money. Actually I had a lot of clients (lawfirms mostly) using these guys when I was doing Microsoft-based consulting. It's a good thing to put in front of Exchange.</comment>


Number of deaths has been outside the IHME prediction intervals 70% of the time - <AUTHOR> https://www.medrxiv.org/content/10.1101/2020.04.11.20062257v1 <comment><USER2> I had to read Table 1 like 10 times to understand what it says. It's an horrible way to present the data.

The Figure 2 is strange. It's a graph of the error in the forecast for 1 day and the error in the forecast for 2 days. It´s not surprising that is is correlated. I´m not sure if it's clear why it is so linear, but perhaps it is necesary to read the underlaying model to undestand that. Anyway, if it is so linear, perhaps it may be useful to use as a empiric correction.

I should undust my probability books, but IIRC the number of cases should follow a Poisson distribution, with a different lambda each day. In that case the 95% interval should be huge, like from almost 0 to 3lambda. It looks difficut to miss an interval that is so huge.</comment>