NIST Reveals 26 Algorithms Advancing to the Post-Quantum Crypto 'Semifinals' - <AUTHOR> https://www.nist.gov/news-events/news/2019/01/nist-reveals-26-algorithms-advancing-post-quantum-crypto-semifinals <comment><USER2> This talk by djb and Tanja Lange is a good overview of the NIST candidates and the current state of pqcrypto: [https://www.youtube.com/watch?v=ZCmnQR3_qWg](https://www.youtube.com/watch?v=ZCmnQR3_qWg)

<comment><USER3> this is a must-watch. It puts the self-congratulatory, hyperbolic & optimistic wording of the article headline into much needed perspective.

<comment><USER4> I'm sorry, what about this headline is "hyperbolic" or "self-congratulatory"? There are 26 algorithms, and they are advancing to another round of consideration. Or are you referring to the claim of "post-quantum"?

<comment><USER3> I was not referring to PQ but to the headline and content of the article which is written like they achieved something and now deserve a pat on their shoulder ...

phrasing like "NIST _reveals_ " doesn't quite reveal the huge actual shit they're in and their failure to make progress on this. It might even suggest that having such a huge number (26) is in anyway positive (instead of admitting to their incompetence wrt narrowing the number to a manageable few).

Seriously watch DJB/TL's video for some background on what is actually happening.

</comment></comment></comment></comment><comment><AUTHOR> The names of the 26 candidates can be found at:

[https://csrc.nist.gov/news/2019/pqc-standardization- process-...](https://csrc.nist.gov/news/2019/pqc-standardization-process-2nd- round-candidates)

<comment><USER5> "For the 2nd round candidates, NIST will allow the submission teams the option of providing updated specifications and implementations (i.e. "tweaks"). The deadline for these tweaks will be March 15, 2019. We originally planned that submission teams would have more time, however recent events out of our control have altered the timeline."

Anyone know what "recent events" refers to?

<comment><USER6> Maybe the federal shutdown?

[https://www.secureworldexpo.com/industry-news/nist- governmen...](https://www.secureworldexpo.com/industry-news/nist-government- shutdown)

<comment><USER5> So the original plan was to announce this earlier then?

<comment><USER7> I believed they were supposed to be announced at RWC in early January. [https://quantumcomputingreport.com/news/nist-to-announce- rou...](https://quantumcomputingreport.com/news/nist-to-announce-round-2-pqc- candidates-on-january-10-2019/)

</comment></comment></comment></comment></comment><comment><USER8> Classic McEliece is 40 years old and still kicking.

The problem is the key size: 512 kb for normal use, 8.5 Mb against quantum computers. On the other hand, 1 MB is not that much if you really want the security.

<comment><USER9> I think any such algo is a non-starter and not very useful.

If we learned anything from the past it's that crypto works and can be deployed if it integrates without users noticing much and if the computational and speed overhead is small (or negative, which in the HTTPS case it often is).

<comment><USER10> Why is the computational and speed overhead sometimes negative for HTTPS?

<comment><USER9> There are performance features that are impractical to deploy unencrypted, so browsers only enable them on HTTPS. (Notably: HTTP/2 and Brotli.)

There are plenty of middlebox devices on the Internet that will drop your traffic if you send anything HTTP over them that they don't understand. So you're kinda "locked in" in not deploying new protocol features. Encryption provides a way out of this.

</comment></comment></comment></comment><comment><USER11> The scary thing about this is that all current HTTPS encrypted traffic is not quantum proof, meaning that if you put it into long-term storage, you'll be able to decrypt the contents at a later point in time, if quantum computers will get available. By then your contents might still be valuable or even have risen in worth.

<comment><USER12> No encryption is intended to be permanent. That's an all-but-impossible feat. Encryption exists to protect data for a meaningful length of time. A threat model extending far into the future is naïve.

The Caesar cipher worked until it didn't, the Enigma was secure until it was broken, and DES was good until it wasn't.

<comment><USER13> I think the parent is trying to highlight an attack vector that most people are unaware of. Most people are happy to see the green lockbox and do not know that it's possible for their browsing habits to be cracked in 30 years when they run for congress.

<comment><USER11> Yeah, that's the point that I wanted to make. Doubt that you want someone to reveal the private sexting images you sent thirty years ago before you even thought about wanting to run for governor or similar. Content like that still has value decades down the line, but it's only protected with encryption that might be considered offline-attackable at a certain point in time. This affects browsing habits as well as end-to-end stuff like Snapchat, Whatsapp, Signal, ...

</comment></comment></comment></comment><comment><USER14> It is indeed prudent to work on post-quantum cryptography now, but no, there's absolutely no quantum computer factoring on the horizon. Any viable path to it, by itself, would be a breakthrough.

<comment><USER15> True, but as we've seen with RSA et al, algorithms once chosen persist for a long time. It would take time for winning algorithms to trickle down to the libraries, and then time for them to be included in the distros. So, better to get going on this stuff now!

<comment><USER16> > but no, there's absolutely no quantum computer factoring on the horizon

Recall the history of ECC. ECC was proposed in the late-1990. Unlike RSA, we already had a solid understanding of its security properties at that time. However, it took 15 years to actually implement and deploy NIST curves on the Internet. And again, another 5 years to deploy a more robust version, Curve25519.

Also, there are enough reasons to believe that the NSA and other major agencies around the world have a storage capacity of several hundreds of exabytes. Signal traffic, Tor traffic or OpenPGP traffic today is already being recorded and kept indefinitely, waiting for a usable quantum computer to decrypt them in the future.

BTW, here's a fun fact: although cryptographers are still trying, but there is no known exploitable structure in most symmetric encryption algorithms (such as ChaCha20 or AES) or hash functions. The only known attack is the general attack using the Grover's Algorithm. It means, if you are afraid that the NSA may decrypt your message in the future, you can always use 256-bit symmetric encryption with per-shared keys. For example, WireGuard supports using pre- shared keys to encrypt ECC handshakes as a poorman's version of PQC.

<comment><USER17> That's why I pushed Merkle Trees back in the day when the topic came up. There's been work extending them in recent years, too.

<comment><USER16> Glad to see you again on HN, thanks for your criticism about the security mitigation techniques previously.

Recently Merkle Trees have indeed made significant progress. XMSS is already standardized, has an RFC, and my Linux distro is already shipping OpenSSH with XMSS authentication enabled. But I don't think it's something practical that you want people to start using immediately.

First, the threats to digital signature created by a quantum computer is not as serious as public-key encryption. If the NSA has intercepted my ECDH handshake, they can recover my session key in the future. But if, by 2045, the NSA has managed to forge my signing key, they cannot use it to log into my server, or create false OpenPGP-signed statements. Either the server has been taken offline for a long time, or at least key has already been retired. Unless it's a digital contract of paramount importance (Excluding blockchains, do we really have digital contracts with a validity of more than 20 years?), I don't think it needs post-quantum capabilities today.

Second, XMSS and many other Merkle Trees-based system has a serious limitation - statefulness. If the private key is not updated correct after each signature, and the state is ever reused, the private key is exposed for everyone to forge. It means, starting using XMSS today creates more hazards than not using it.

I do see some reasons to standardize XMSS: (1) Unlike public-key encryption schemes, the security properties of Merkle Trees-based signature schemes are well-known, with strong guarantees. (2) Standardizing it early allows people to be familiar with the standardization process, eases the future process of more PQC standardization, (3) Allow people to implement and deploy it in an experimental setup.

But I really wished to see the standardization of a stateless hash-based signature, like djb's SPHINCS-256.

I'm curious about your takes on this topic.

<comment><USER17> "Glad to see you again on HN"

You as well. I bookmarked your rebuttal to dismissing NIST by default. I had one but yours is way better. I'll just drop that link from now on. :)

"XMSS is already standardized, has an RFC, and my Linux distro is already shipping OpenSSH with XMSS authentication enabled. "

I've been away from crypto for a while. I had no idea this all happened. Thanks for the tip!

"the threats to digital signature created by a quantum computer is not as serious as public-key encryption."

For public-key encryption, I was recommending using several together such as today's strongest plus NTRU and/or McEliece. At the least, one that's strong in classical model plus one that's strong in quantum model. This is for people concerned about governments, large corporations, and organized crime (including cartels). These are the groups most likely to be able to afford and willing to use such codebreaking capabilities on their opponents. Damage can happen even a long time later. That's why U.S. government defaults on around 40 years before declassification of TS data to water down the effects of release.

For signatures, a lot has changed since I last looked at it. The Certificate Transparency schemes might be combined with Merkle schemes in interesting ways. There's also lots of roll-outs on Git/Github which is versioned. People are using smart contracts and blockchains. Then, there's lots of legacy tech with the CA's and DNS system. I think any new ideas about how to develop these alternative techs and get them adopted should be designed to fit within what's going on. I'd probably have to think deeply about all that before I could give you fresh answers on signatures. I'll definitely write it up on these threads if I ever do. Right now, I'm still more focused on getting medium-to-high assurance of design/implementation more cost-effective.

</comment></comment></comment></comment></comment></comment><comment><USER18> reminder that NIST should not be trusted given their history with regards to standards around vulnerable crypto as well as potential for manipulation due to "higher level" .gov organizations:

[https://projectbullrun.org/dual- ec/vulnerability.html](https://projectbullrun.org/dual-ec/vulnerability.html)

<comment><USER19> > reminder that NIST should not be trusted given their history with regards to > standards around vulnerable crypto as well as potential for manipulation due > to "higher level" .gov organizations:

And, you also need to remember that NIST seems to have known about "differential cryptanalysis" before everybody else and made DES actually _more_ resistant to it.

"Trust, but verify" should _always_ be the rule.

<comment><USER20> NIST didn't do that. The NSA did.

NIST submitted the AES candidates to the NSA for feedback and the NSA proposed S-box changes without explaining why.

Turns out hose changes made AES more resilient to differential cryptography.

</comment></comment></comment><comment><USER21> Dupe: [https://news.ycombinator.com/item?id=19037286](https://news.ycombinator.com/item?id=19037286)</comment>


A Recipe for Growth: Adding Layers to the Cake - <AUTHOR> http://jeff.a16z.com/2012/01/18/a-recipe-for-growth-adding-layers-to-the-cake/ <comment><USER2> Now that is good stuff. I mean, it seems obvious in a sense, and arguably it distills down to "grow by offering a continually expanding portfolio of complementary products." But for folks who haven't spent a lot of time thinking about growth strategies, or who have tunnel vision over their current product, this article could be a real eye-opening.

At Fogbeam Labs, we use the analogy of an onion, with it's many layers, instead of a cake analogy... but the point is the same. Know exactly what you're doing, and have great focus; but also have a strong awareness of what your complements as a good leading indicator of where you can grow.

BTW, on that note... for the hackers in the crowd, who are doing startups but have never cracked open a marketing book or anything "business related," I'd like to offer a suggestion. Pony up the couple of hundred bucks or so that it'll cost, and commit 1 semester to taking a "Marketing 101" class at the local community college. Or at least buy the book and read it. Despite what you might believe, marketing isn't all about hucksterism and selling snake-oil to unsuspecting rubes.</comment>


Ask HN: Is the Tor Browser Developers GPG Key Attacked or Compromised? - <AUTHOR> When trying to download the Tor Browser Developers key to verify the download (as per https://support.torproject.org/tbb/how-to-verify-signature/), two working public key servers I know of return a 24.8 Megabyte file with over 26 million characters:<p>> https://keys.gnupg.net/pks/lookup?op=get&search=0x4E2C6E8793298290<p>> https://pool.sks-keyservers.net/pks/lookup?search=0x4E2C6E8793298290&fingerprint=on&op=index<p>Surely this can't be right. In fact, I'm worried about even trying to import them into GPG. Has someone vandalized or otherwise broken the signing keys? <comment><USER2> Probably related to this:

[https://nakedsecurity.sophos.com/2019/07/05/openpgp- experts-...](https://nakedsecurity.sophos.com/2019/07/05/openpgp-experts- targeted-by-long-feared-poisoning-attack/)

[https://www.zdnet.com/article/openpgp-flooded-with-spam- by-u...](https://www.zdnet.com/article/openpgp-flooded-with-spam-by-unknown- hackers/)

I'm sure I read in that last day or two about software updates to mitigate against this, but I can't find anything now.

There is a 2 day old release of GnuPG available - which isn't described on their release notes page (yet?):

[https://gnupg.org/download/index.html](https://gnupg.org/download/index.html)</comment>


600k concurrent websocket connections on AWS using Node.js (2015) - <AUTHOR> https://blog.jayway.com/2015/04/13/600k-concurrent-websocket-connections-on-aws-using-node-js/ <comment><USER3> A few pieces of advice based on running [https://github.com/mozilla- services/autopush-rs](https://github.com/mozilla-services/autopush-rs), which handles tens of millions of concurrent connections across a fleet of small EC2 instances.

1) Consider not running the largest instance you need to handle your workload, but instead distributing it across smaller instances. This allows for progressive rollout to test new versions, reduces the thundering herd when you restart or replace an instance, etc.

2) Don't set up security group rules that limit what addresses can connect to your websocket port. As soon as you do that connection tracking kicks in and you'll hit undocumented hard limits on the number of established connections to an instance. These limits vary based on the instance size and can easily become your bottleneck.

3) Beware of ELBs. Under the hood an ELB is made of multiple load balancers and is supposed to scale out when those load balancers hit capacity. A single load balancer can only handle a certain number of concurrent connections. In my experience ELBs don't automatically scale our when that limit is reached. You need AWS support to manually do that for you. At a certain traffic level, expect support to tell you to create multiple ELBs and distribute traffic across them yourself. ALBs or NLBs may handle this better; I'm not sure. If possible design your system to distribute connections itself instead of requiring a load balancer.

2 and 3 are frustrating because they happen at a layer of EC2 that you have little visibility into. The best way to avoid problems is to test everything at the expected real user load. In our case, when we were planning a change that would dramatically increase the number of clients connecting and doing real work, we first used our experimentation system to have a set of clients establish a dummy connection, then gradually ramped up that number of clients in the experiment as we worked through issues.

<comment><USER4> Could you explain or post a link to something about #2? I've never heard of that before!

<comment><USER3> I can not. AWS, for whatever reason, does not want this publicly documented. I would write up what we learned in our testing in more detail, but I have been asked not to.

You can find some discussion of this behavior in places like [https://forums.aws.amazon.com/thread.jspa?threadID=231806](https://forums.aws.amazon.com/thread.jspa?threadID=231806). I originally became aware of the issue, before hitting in production, from the HN comment at [https://news.ycombinator.com/item?id=18314138](https://news.ycombinator.com/item?id=18314138)

<comment><USER5> I'm glad you found my comment helpful!

I myself found out about to avoid the connection tracking in this thread [https://news.ycombinator.com/item?id=15724072](https://news.ycombinator.com/item?id=15724072)

It's super frustrating that Amazon doesn't document this in a more approachable manner.

</comment></comment></comment></comment><comment><USER6> Some dude in 2012 did 1 million active connections with whatever node.js and v8 versions we had at that time :) [http://blog.caustik.com/2012/08/19/node- js-w1m-concurrent-co...](http://blog.caustik.com/2012/08/19/node- js-w1m-concurrent-connections/)

Edit: I wasn't aware but there was some build-up to it: \- 100k connections: [http://blog.caustik.com/2012/04/08/scaling-node-js- to-100k-c...](http://blog.caustik.com/2012/04/08/scaling-node-js- to-100k-concurrent-connections/) \- 250k connections: [http://blog.caustik.com/2012/04/10/node- js-w250k-concurrent-...](http://blog.caustik.com/2012/04/10/node- js-w250k-concurrent-connections/)

</comment><comment><USER7> Yeah, we used nginx-push-stream-module[1] to support 1M connections with lower boxes. Websocket-as-a-service. Really cool module. Was a realtime project for a live TV contest where people could participate with their phones.

[1] [https://github.com/wandenberg/nginx-push-stream- module](https://github.com/wandenberg/nginx-push-stream-module)

<comment><USER8> Did not know about this module. Instead, we are using a pretty identical one [1]. Works flawlessly

[1] [https://github.com/slact/nchan](https://github.com/slact/nchan)

<comment><USER7> Sweet, didn't know about nchan too.

</comment></comment></comment><comment><USER9> So 1/4 of what Elixir/Erlang can handle, but more difficult and less reliable:

[https://phoenixframework.org/blog/the-road-to-2-million- webs...](https://phoenixframework.org/blog/the-road-to-2-million-websocket- connections)

<comment><USER10> "but more difficult" sounds pretty subjective... I can get behind the "less reliable" comment because Erlang is rock solid.

Sorry to be that guy, but I can find a TON more people capable of supporting ECMA vs. Erlang and/or Elixr. I say this as a huge fanboy of both.

I know I'll get rolled on HN for saying this because we all drink the optimization Koolaid but I feel this is worth mentioning.

<comment><USER11> You have a fair point, but if by "more difficult" the grandparent meant "more difficult to maintain", I'd have to agree with him. The failure modes of processes running in the BEAM are much more transparent and understandable than the usual ad-hoc JS nodejs server.

<comment><USER10> IMO they're on the same level but I think it's just my style of how I manage my Node.js applications down my try/catch blocks, exception handling, and primarily process isolation.

> usual ad-hoc JS nodejs server

Yep - that's the key why I feel they're on the same level. I heavily rely on the cluster module and IPC to get my work done which gives me true process isolation/safety however I admit it's more code in Node.js to make it rock- solid ;)

Anecdotal, but I find that the typical Node.js developer is 100% "single- process" with weird supervisors like PM2. I also see over-engineered fleets of EC2 instances behind ELB/ALBs with health checking kicking bad instances out + a way to replace them... this is also stupid common in Docker/docker- compose/K8s as well.

For the record I think that development pattern described in the above paragraph is total crap and I put it on the same level of idiots giving modern PHP7 a bad name. Devs who practice like this completely blew the content from their CS100/200 level courses out their butt (yeah yeah I say this as a dropout myself whatever ha).

</comment></comment></comment></comment><comment><USER12> There is more performant web-socket implementation than the one mentioned in the blog. It can handle 6X more connections and much less memory

[https://github.com/uNetworking/uWebSockets.js](https://github.com/uNetworking/uWebSockets.js)

EDIT:

Note that the blog post is from 2015. There are many optimization (Ignition and TurboFan pipeline) has been done in V8 since then, especially offloading GC activity to separate thread than NodeJS Main thread.

<comment><USER13> Unfortunately it is written by someone whose technical ability far exceeds his people skills, which are essential for a library module that developers can safely depend on.

This is doubly unfortunate because I very much share his views on bloated frameworks etc. :(

<comment><USER14> Yeah that's the dev who pushed a purposefully broken version of their library to npm because he was upset that they wouldn't let him delete it.

</comment></comment></comment><comment><USER15> There is that Phoenix thing when they famously did 2 Million.

[https://phoenixframework.org/blog/the-road-to-2-million- webs...](https://phoenixframework.org/blog/the-road-to-2-million-websocket- connections)

I agree with the suggestion that smaller instances that can be scaled is not a bad idea.

</comment><comment><USER16> I don't get point of using Node.js when compared to something like Elixir. Elixir's Phoenix can handle more numbers of concurrent connections as well as provide reliability with better programming abstractions, distribution, pretty good language.

<comment><USER17> This is actually not true, if you take a really optimzied C/C++ library for WS Nodejs will crush Elixir by a large margin. Elixir / Erlang is slow and consume a lot of memories compared to more native languages or C libraries.

Ex: [https://github.com/uNetworking/uWebSockets](https://github.com/uNetworking/uWebSockets)

<comment><USER18> At that point, you may as well implement your own server in C++ using epoll.

<comment><USER19> Nah. Devs don't touch C++ code in this case.

</comment></comment></comment></comment><comment><USER20> Interesting details; it would be nice to see how those ulimit/networking numbers were arrived at.

The title should have [2015].

<comment><AUTHOR> Yep - title should've had [2015], my bad!

<comment><USER21> Added now.

</comment></comment></comment><comment><USER22> Genuinely missing the point question: isn't the number of concurrent socket (websocket or otherwise) connections just a function of the underlying OS and number of instances thereof, not a function of Node.js ?

<comment><USER23> Even if the OS can take it, it requires proper engineering in your stack (ie, node.js here).

it wasn't long ago that even managing 10K connections on a server was considered quite a feat - <USER2> [http://www.kegel.com/c10k.html](http://www.kegel.com/c10k.html)

</comment></comment><comment><USER24> Doesn't sound so impressive. I've done close to a million on a single Digital Ocean droplet using nchan[0] before. Latency was reasonable even with that many connections, you just need to set your buffer sizes carefully. Handshakes are also expensive, so it's useful to to be able to control the client and build in some smart reconnect/back-off logic.

[0] [https://www.nginx.com/resources/wiki/modules/Nchan/](https://www.nginx.com/resources/wiki/modules/Nchan/)

<comment><USER25> would be nice to compare all providers with something like [https://www.techempower.com/benchmarks/](https://www.techempower.com/benchmarks/) (azure & aws)

</comment></comment><comment><USER26> Does anyone have a more recent experience? I currently use socket.io 2.2 with node v10.16, no v8 tweaks in a docker container. At ~1000 sockets, sometimes the server receives spikes of 8000 HTTP reqs/sec, which it has to distribute to the websockets, up to 100 msgs/sec, ~1kb/msg to each socket. These spikes are making the server unstable, socket.io switches most of the clients from websockets to xhr polling.

<comment><USER27> I don't have any experiments to share, but you can go father if you stop using socket.io, but I guess you need something to deal with long polling.

You should consider tweaking --max_old_space_size, we got a lot of mileage giving node more memory.

<comment><USER26> I make heavy use of socket.io chatrooms and have to support older browsers. Haven't found any better solution yet, but I'll try the tweak, thanks. I'm also looking into load balancing via haproxy or nginx, since I need less than 10k concurrent clients.

</comment></comment></comment><comment><USER28> I thought this was going to be about AWS-managed websockets using API Gateway. I've been using that at a really small scale and it's got a great API but other than almost certainly being much more expensive than the EC2 machine used here I wonder how well it works with that sort of scale.

</comment><comment><USER29> I have written and deployed message queues in Node.js that take data from Redis and push it out on websockets. It is a pain to deal with the GC in weird cases. This was about 5 years ago, so the details might not accurate.

Things worked fine until some day some customer started sending multi-megabyte strings over the system. It is difficult to actually track down that it is GC that is halting the system and then figuring out ways to fix the issue. We ended up not using JavaScript strings and instead using Node.js buffers to do the transport - I don't recall the Node.js library for Redis supporting that out of the box.

<comment><USER19> It's better but still a pain. The GC is so much more fragile than the JVM.

Would have used nchan for that probably :p

</comment></comment><comment><USER30> Does anyone have experience doing the same on GCP?

In particular right now I am trying to add live reloading to my App Engine Standard app but Standard doesn't support long lived connections (so no websockets) and App Engine Flexible seems like it will be pricy.

I think I can set up a single separate websocket instance which is only responsible for doing listen/notify on postgres and telling the client when it's time to refetch from the main webserver again.

Does this sound approximately workable? Will I actually be able to reach the connection numbers like in this article?

</comment><comment><USER31> I wonder how it would compare in terms of cost with doing it via the api- gateway. It would depend on how your app and user base scales and what the sockets are being used for I suppose.

[https://docs.aws.amazon.com/apigateway/latest/developerguide...](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway- websocket-api-overview.html)

<comment><USER27> Api gateway is very expensive once you get sustained load. EC2 much cheaper.

</comment></comment><comment><USER32> Lots of wisdom on this page ;)

Just want to add. Real-world, often the predominant use case is not optimizing for "max-conns". But <100,000 concurrent users, who instead need to be connected for a very long time.

In this instance, I've found Caddy's websocket directive, inspired by Websocketd, to be quite robust and elegant. It's just a process per conn. Handling stdin, stout style messaging ;)

</comment><comment><USER33> But once they all start doing stuff then what happens? 600k is more of a function of memory right?

</comment><comment><USER34> Why are websites still hijacking scroll behavior in 2019? I can't even take the article seriously with my scrolling bouncing and glitching all over the place.

</comment><comment><USER19> A better solution would be to use nchan+nginx and then your Node API is just a simple stateless REST service. Will scale better and be easier to maintain.

</comment><comment><USER35> How do you test something like this internally?

<comment><USER36> Load testing tools like Tsung.

</comment></comment><comment><USER36> So what's the point of attempting to max out the count of idle connections on some cloud engine?

</comment><comment><USER37> I did this with Netty back in 2010. Increasing nf_conntrack_max is a hard- learned lesson.

</comment><comment><USER38> Why is this specific to AWS ?

<comment><USER39> I think the idea is to dispel the myth that you can't get that kind of performance on a cloud instance.

<comment><USER40> From what I understand there is no real "performance" in said test. Just a bunch of idle connections doing big nothing and eating all CPU time with garbage collection.

</comment></comment></comment><comment><USER41> Does anyone know what latency overhead switching to ECS/EKS would add?

</comment><comment><USER42> It's not difficult to see that such amount of connections is only useful if your server performs a trivial task. Otherwise, the bottleneck will be your CPU(s).

</comment><comment><USER43> What do you need this many connections for?

<comment><USER44> One possible use is for high traffic websites. For example, StackExchange peaked at 500k concurrent websocket connections back in 2016. [0]

[0] [https://nickcraver.com/blog/2016/02/17/stack-overflow-the- ar...](https://nickcraver.com/blog/2016/02/17/stack-overflow-the- architecture-2016-edition/#websockets-httpsgithubcomstackexchangenetgain)

<comment><USER45> Surely not on one endpoint

<comment><USER46> We're spread over 9 servers but only due to ephemeral port and handle exhaustion issues. Each server is fronted by 4 HAProxy frontends that each handle ~18k connections.

Since Nick's post we've moved from StackExchange.NetGain to the managed websocket implementation in .NET Core 3 using Kestrel and libuv. That sits at around 2.3GB RAM and 0.4% CPU. Memory could be better (it used to be < 1GB with NetGain) but improvements would likely come from tweaks to how we configure GC under .NET Core which we haven't really investigated yet.

We could run on far fewer machines but we have 9 sitting there serving traffic for the sites themselves so no harm in spreading the load a little!

<comment><USER5> Ephemeral port exhaustion is easy to handle if you control HAProxy and the origins.

You'll need the source [1] option on your server lines, and you also need to adjust to allow more connections, one of these will do: have the origin server listen on more ports, add more ips to the origin server and listen to those too, add more ips to the proxy and use those to connect as well.

I'm not sure about handle exhaustion? I've run into file descriptor limits, those are usually simple to set (until you run into a code enforced limit)

[1] [https://cbonte.github.io/haproxy- dconv/1.7/configuration.htm...](https://cbonte.github.io/haproxy- dconv/1.7/configuration.html#5.2-source)

<comment><USER46> I might be missing some details there to be honest; I'm more familiar with the application side of things than the infrastructure :)

</comment></comment></comment></comment></comment></comment><comment><USER47> Would be nice to rebench with the new websocketstream API!

[https://github.com/ricea/websocketstream- explainer/blob/mast...](https://github.com/ricea/websocketstream- explainer/blob/master/README.md)</comment>


An Introduction to the Central Limit Theorem - <AUTHOR> http://spin.atomicobject.com/2015/02/12/central-limit-theorem-intro/

<comment><USER2> > Often referred to as the cornerstone of statistics

Well... often referred to as the central theorem of statistics. Each time you say its name. What's central is the theorem, not the limit. It was Polya who first called it that, "zentraler Grenzwertsatz".

> Why the Central Limit Theorem Works

Well... I don't think that's really an explanation at all of why e^(-x^2/2) is such a privileged function. Why would any distribution converge to a normal distribution?

It essentially boils down to the Fourier transform. When you take the Fourier transform of the sample means, if you ignore all but the quadratic terms (there are no linear terms if you centralise to mean 0 and variance 1), you get the exponential limit (1 - t^2/2n)^n. That's the Gaussian, which is its own Fourier transform.

[https://en.wikipedia.org/wiki/Central_limit_theorem#Proof_of...](https://en.wikipedia.org/wiki/Central_limit_theorem#Proof_of_classical_CLT)

In other words, because the Gaussian is its own Fourier transform, sample means converge to the Gaussian.

<comment><USER3> > _Well... I don 't think that's really an explanation at all of why e^(-x^2) > is such a privileged function. Why would any distribution converge to a > normal distribution?

> It essentially boils down to the Fourier transform_

Precisely.

One motivation for taking a fourier transform is that the the p.d.f. of the sum of several random variables is the convolution of corresponding p.d.fs, and that convolution in the original domain is equivalent to multiplication in the fourier domain (by the convolution theorem).

<comment><USER4> ...and p.d.f's convolve when you derive the distribution of a random variable defined as sum of other independent random variables.

</comment></comment></comment><comment><USER5> I always liked this visual representation of the central limit theorem: [http://blog.vctr.me/posts/central-limit- theorem.html](http://blog.vctr.me/posts/central-limit-theorem.html). There is a faster one here (I think written in R): [http://vis.supstat.com/2013/04/bean- machine/](http://vis.supstat.com/2013/04/bean-machine/)

These are computer simulations of Galton boxes: [http://en.wikipedia.org/wiki/Bean_machine](http://en.wikipedia.org/wiki/Bean_machine)

<comment><USER6> As the comments to that page mention, though, it's not a representation of the central limit theorem at all. It shows how a binomial distribution "becomes" a normal distribution in the long run, but it doesn't show why this would apply to the distribution of errors of a series of sample means.

</comment></comment><comment><USER7> I think the first half of the article showing how this works with a given sample distribution is pretty good. I don't think it's really doing much to build intuition at the end, though.

It's also worth pointing out that there are distributions for which the central limit theorem doesn't hold (e.g. the sum of samples from a Lorentzian distribution will again be Lorentzian, not Gaussian.)

<comment><USER2> > the sum of samples from a Lorentzian distribution will again be Lorentzian, > not Gaussian

Lorentzian? I had to look that up. Oh. Cauchy distribution. Right, because it doesn't have any finite moments, because the tails are too heavy.

<comment><USER8> Physicists like to say Lorentzian (or sometimes Breit-Wigner) instead of Cauchy.

</comment></comment></comment><comment><USER9> I have a series of basic questions I include in any data science interview, and one is "please describe what the central limit theorem says in simple, high-level terms". It's absolutely amazing how many people who have great credentials can't do this. I get a lot of "any distribution becomes normal when you sample it enough". This is nonsensical and shows a lack of understanding of the theorem.

Please, if you claim to know stats, understand what the central limit theorem says. It's a pretty incredible and useful theorem.

<comment><USER10> Out of curiosity, mind sharing some of those questions? I'm a computer science major looking into getting a data science internship, and I'd love for a chance to see some real interview questions.

<comment><USER9> Yep! Here are a few, I customize the questions a bit based on seniority and background.

1\. In a 1-d gaussian with mean=0 and variance=1, what is p(x=0)?

2\. If I give you a csv file with a header, how would you return a tsv file with columns 6 and 2? Just high-level, not actual implementation.

3\. If I have some dataset (I change it up a lot, a common one is subway ridership by station and date), how would you create a visualization to give some insight to a fairly non-technical audience, e.g. the distribution of ridership by station.

4\. If I gave you a house dataset with a bunch of attributes (e.g. year built, number of sq feet, presence of pool, zip code), how would you go about building a machine learning model that predicts whether the house is being rented or is owned (assuming you are provided with tagged data)? The selling price? How many times it has been sold?

5\. What's a technology that you're excited about (could be a computing tool, a machine learning algorithm in a library, etc)? Tell me some of the pros and cons of it and competing technologies.

We also give out a 4-6 hour task as homework, and evaluate the code quality, tools used, answers provided, etc.

</comment></comment></comment><comment><USER11> My introduction to the central limit theorem was that chained independent random processes tend to result in a Gaussian distribution. This is so general that one is surprised when one finds non-Gaussian distributions (canonical example: the stock market).

I attended a lecture by Mandelbrot (shortly before he died) where he spoke at length about this- take a look at stable distributions and the generalized central limit theorem.

<comment><USER12> I think that Mandelbrot's point is actually quite the opposite - that non- gaussian stable distributions are the norm; and that we are trained to see gaussians and that's why we're surprised. When alpha is very close to 2 (e.g. 1.9 and above it really makes little difference) but even a bit lower like 1.8 you can have some very scary consequences for modelling. In his book he has an example of an insurance market and modelling the failure of insurance companies... Which was especially poignant considering what happened after he died (and what is happening now).

</comment></comment><comment><USER13> Is it not tradition to use n for sample size rather than N? N is typically population size (in my experience).

<comment><USER2> I don't think any tradition is too entrenched on this particular point.

<comment><USER13> You might be right. In my stats courses, we used lower case letters for all sample estimates and upper case for all population values.

</comment></comment></comment><comment><USER14> To me.. the core idea is that (given one chooses, over and over, from a bunch of independent and identically distributed events.):

There are more ways for everything to happen than there are ways for one thing to happen over and over.

</comment><comment><USER15> An interesting bit of trivia for computer history buffs:

Alan Turing independently discovered the Central Limit Theorem while still an undergrad in 1934.</comment>


150+ Lesser Known Growth Hacking Strategies - <AUTHOR> http://growthhackingexamples.com/growth-hacking-strategies/ <comment><AUTHOR> how do u find the collection?</comment>


A generic beer that was too cheap for Saskatchewan - <AUTHOR> https://www.cbc.ca/archives/the-generic-beer-that-was-too-cheap-for-saskatchewan-1.5064323?cmp=rss <comment><USER2> The state of liquor/beer sales in Canada is sad indeed. In BC I can drive 5 hrs to the US and buy beer that is made 5 minutes from my house for less than half the cost.

That same brewery gives every employee free beer every week because that is cheaper than giving employee discounts.

And now legal cannabis pricing is even worse.

<comment><USER3> I had never heard of keeping your receipt with your beer until visiting a liquor store in Alberta. Apparently its common to confiscate beer lacking a receipt, as its assumed to be from another province? Seems very silly, but my perspective is skewed as a Seattleite ($25 fine for drinking in public that is rarely enforced).

<comment><USER4> I have never heard of this either, but yeah... through a quirk in Canadian law, only the crown is allowed to transport liquor between provinces, modulo some exceptions for small quantities: [https://constitutional.findlaw.ca/article/how-much- alcohol-c...](https://constitutional.findlaw.ca/article/how-much-alcohol-can- i-bring-from-another-province/)

I can't legally very much beer or wine back to Saskatchewan when I make a trip to Alberta, even though it's significantly cheaper there.

Edit: I knew about the "no transport between provinces" thing, but I didn't know about the receipt thing. I suspect that if you're in Alberta, with Alberta plates, you're not going to have your beer confiscated. If you're in Saskatchewan with Alberta plates, and have a trunk full of beer... they're probably going to suspect you bought it all cheap at home and are bringing it to friends.

<comment><USER5> >only the crown is allowed to transport liquor between provinces

Coincidentally, that restriction was just removed! The Minister for Intergovernmental Affairs announced it today. [1]

However, provinces and territories can still legislate their own barriers. But federally, the law no longer stands.

[1] [https://www.canada.ca/en/intergovernmental- affairs/news/2019...](https://www.canada.ca/en/intergovernmental- affairs/news/2019/04/canada-acts-to-eliminate-barriers-to-interprovincial- trade-in-alcohol.html)

<comment><USER4> Holy crap! I bet that's why this article got dredged up. That's really great. Thanks for the heads up!

</comment></comment></comment></comment></comment><comment><USER6> Most provinces have a minimum price threshold (e.g. [https://www.ontario.ca/laws/regulation/100116/v2](https://www.ontario.ca/laws/regulation/100116/v2)), and then the tax is a major component of that retail price -- something like 47% for beer, as a so-called sin tax.

As an occasional consumer, Ontarian and Canadian, I don't mind. Alcohol has a lot of societal costs.

<comment><USER7> Agreed. While I find minimum price thresholds and sin taxes personally annoying, there's lots of good research associating ease of access and low prices with alcoholism, which has high costs in terms of healthcare and individual suffering.

<comment><USER8> I hate it. It just more of Canadian government paternalism, taking away people's agency: "you cannot be trusted to make the right decision, so we'll make the wrong decision more expensive to do." The worst thing isn't even the paternalism, it's the hypocrisy. "This tax is to protect you and everyone around you" but in reality "oh yeah please don't stop drinking because it's bringing in so much tax revenue."

<comment><USER9> Yes, that's what libertarian paternalism (see: Nudge by Thaler and Sunstein) is meant to do. Legal and abundant alcohol imposes high negative externalities on the rest of the population. Sin taxes reduce consumption because economic incentives usually work.

It's when things are outright illegal (see: prohibition, narcotics) or the cost of ignoring the law is _de facto_ lower than the legal price that things become quite dangerous for consumers, like the sale of narcotics laced with cheaper, more potent compounds like fentanyl.

For better and worse, a regulated market that pushes prices upward is usually better than an unregulated market or a criminalized one.

<comment><USER10> >libertarian paternalism

Just when you think you've seen everything; someone actually came up with this term? Is this meant to try to convince libertarians as to the merits of statism and regulation?

</comment></comment></comment></comment></comment><comment><USER11> This largely came about because of the "no name" craze started by Loblaws that was going on during this time [0]. Plain yellow packaged products with generic naming and no pictures, priced low, and of reasonable quality. Because when all you wanted was "Raisin Bran", why pay more? (Yeah, well, to get more raisins, but that's beside the point). I had a lot of yellow groceries in my apartment back in my starving student days.

[0] [https://en.wikipedia.org/wiki/No_Name_(brand)](https://en.wikipedia.org/wiki/No_Name_\(brand\))

<comment><USER12> That brand is disgustingly ironic.

<comment><USER13> Better than ironically disgusting.

</comment></comment></comment><comment><USER14> How much did it cost??</comment>


Ask HN: What Are Some Good African, Asian, South American News Websites? - <AUTHOR> I'm constantly looking to build and increase my knowledge of the world, but I realised the 'World' section of many news websites I follow (NYTimes, Wash Post, Guardian, BBC, FT, WSJ etc) do not have a particular large amount of African, Asian, South American news.<p>What news websites do you guys know/recommend to follow for news in these countries? Language would preferably be in English. <comment><USER2> Al Jazeera should definitely be on your list for non-Western coverage: [https://www.aljazeera.com/news/](https://www.aljazeera.com/news/)

Their coverage of the middle-East is likely to be biased by their Qatari ownership, but they cover major world stories neglected by the Western press. More details about their potential bias here: [https://en.wikipedia.org/wiki/Al_Jazeera_controversies_and_c...](https://en.wikipedia.org/wiki/Al_Jazeera_controversies_and_criticism).

<comment><USER3> Extremely one sided and pro-islamic.

<comment><USER4> I'm not sure what you mean? Of course, there's an Islamic bias because it's written by Muslims. But it's a lot fairer than most media.

<comment><USER3> Not just bias, but sometimes a propaganda piece. One side of the story, often the most significant one is left out.

It is funded by the state of Qatar to act as its mouthpiece. In the Islamic world every country legitimizes itself by trying to sound more islamic than the other.

I am just stating that they may be strongly biased. They do make crisp documentaries and interesting news stories.

<comment><USER5> Could you link a biased story?

I would put their bias on par with CNN, but would like a story to see what you mean. What is a good example?

<comment><USER6> [https://www.memri.org/tv/al-jazeera-holocaust-denial- israel-...](https://www.memri.org/tv/al-jazeera-holocaust-denial-israel- biggest-winner-same-justification-annihilate-palestinians/transcript)

here is an article about a video they published which they took down after enough backlash. imagine what articles / videos were just as biased but didn't get taken down

</comment></comment></comment></comment></comment></comment><comment><USER7> [https://www.project-syndicate.org/](https://www.project-syndicate.org/) because its more from the source of international figures, a bit like an Twitter for intelligent international scholars and public figures.

</comment><comment><USER8> For Asian regional news in English I'd recommend Southern China Morning Post. It's based out of Hong Kong and still independent from the CCP's grasp (for now).

<comment><USER9> SCMP was created by "western" anti-chinese propagandists.

"The journal was founded by Australian-born anti-Qing dynasty revolutionary Tse Tsan-tai and British journalist Alfred Cunningham"

[https://en.wikipedia.org/wiki/South_China_Morning_Post](https://en.wikipedia.org/wiki/South_China_Morning_Post)

It's strange you think it is something to be recommended pre-alibaba, but not something to be recommended post-alibaba.

<comment><USER10> How is Tse Tsan-tai a western, anti-chinese propagandist? He was a collaborator w/ Sun Yat-sen. I realize the two drew influence from western ideals (and christianity), but wonder what makes them anti-chinese.

That the Chinese government praised Sun Yat-sen, and printed a commemorative coin for his 150th birthday, seems at odds with your sentiment.

[http://www.chinadaily.com.cn/china/2016-10/25/content_271713...](http://www.chinadaily.com.cn/china/2016-10/25/content_27171381.htm)

<comment><USER9> > How is Tse Tsan-tai a western, anti-chinese propagandist?

"Born in Grafton, New South Wales, to a "patriotic" Chinese, Tse Yat-cheong (謝日昌), Tse Tsan-tai was baptised James See on 1 November 1879."

[https://en.wikipedia.org/wiki/Tse_Tsan- tai](https://en.wikipedia.org/wiki/Tse_Tsan-tai)

When I wrote anti-chinese, I meant at the time - qing china. And the SCMP was created by a british "journalist" and Tse specifically to push anti-chinese propaganda at that time.

> That the Chinese government praised Sun Yat-sen, and printed a commemorative > coin for his 150th birthday, seems at odds with your sentiment.

Why? The CCP can choose to celebrate parts of sun yat-sen's legacy which aligns with their own ideology - especially the destruction of china's feudal system. Also, you can praise sun yat-sen as a larger part of reincorporating taiwan back to the mainland.

We celebrate Robert E. Lee, does it make the US anti-US? Hell in the 1970s, the US celebrate Mao and Deng in order to grow our relationship. Doesn't make the US pro-communist.

<comment><USER10> > When I wrote anti-chinese, I meant at the time

Ah, okay. I read your original post as critical of scmp as anti Chinese today, but am not sure if that's what you meant now.

</comment></comment></comment></comment></comment><comment><USER11> The Intercept and Democracy Now cover South America quite a bit.

Particularly politics and environmental issues, for example Glenn Greenwald lives and has been reporting in Brazil.

Some good advice someone gave me awhile ago was to follow journalists themselves instead of news outlets

</comment><comment><USER12> I would recommend the Economist for well-written world news. It's not free, but well worth paying for. [https://www.economist.com/](https://www.economist.com/)

<comment><USER3> They do have free podcasts. Their bias is colonialist in nature.

<comment><USER13> Rather than criticize the suggestions of others in this thread, how about provide some suggestions that meet your own criteria? And note that you always have to consider the source. At the very least, people are human: they're not omniscient, write for a particular audience, and have limited resources. These in and of themselves contribute to bias without any malicious or underhanded intent.

<comment><USER3> I just presented my opinion. I am a regular listener. As a non westerner I know for a fact that stories about my own country is dominated by a colonial view point.

The OP is specifically looking for sources that do not have western bias. And I hope my opinion will be useful.

</comment></comment></comment></comment><comment> w-ll I'm a big fan of NHK. This is their news, but they have some other programming thats kinda similar to PBS [https://www3.nhk.or.jp/nhkworld/en/news/](https://www3.nhk.or.jp/nhkworld/en/news/)

</comment><comment><USER14> [https://thediplomat.com/](https://thediplomat.com/)

^IMO hands down the best site for geopolitical editorial coverage of Asia.

</comment><comment><AUTHOR> Thank you! For all the suggestions, I'll definitely be adding all of these to my list. Any more suggestions is greatly appreciated!

</comment><comment><USER15> In South Africa, the Daily Maverick and the Mail & Guardian. Both more politics than breaking news, both left-leaning, both well-regarded.

</comment><comment><USER16> If you want actual coverage I would go to bellingcat

</comment><comment><USER17> [https://asia.nikkei.com/](https://asia.nikkei.com/) from Japan is quite good.

</comment><comment><USER18> Al Jazeera [https://www.aljazeera.com/](https://www.aljazeera.com/)

</comment><comment><USER19> for news about India, you can use The Hindu. Generally, the articles tends to be neutral to left-leaning.

[https://www.thehindu.com/](https://www.thehindu.com/)

</comment><comment><USER20> Checkout this YC company from the W19 batch [https://www.thejuggernaut.com/newsletter](https://www.thejuggernaut.com/newsletter)

</comment><comment><USER21> Bloombergs different regional sites, eg. [https://www.bloomberg.com/middleeast](https://www.bloomberg.com/middleeast)

</comment><comment><USER9> > I follow (NYTimes, Wash Post, Guardian, BBC, FT, WSJ etc)

What's the point? They are all essentially the same news company. They push the same narrative and agenda. Just pick one and move on with your day.

The problem is that a lot of african, asian and south american news companies are created by, owned by or run by by westerners. And most of them, like the SCMP, were created by the west to push western propaganda. So essentially, you aren't getting african, asian and south american news. You are getting western propaganda about african, asian and south american news.

Also, africa, asia and south america are huge continents. I don't think there is such a thing called "african, asian and south american" news.

For geopolitical news, xinhua/cgtn for china, rt for russia, al jazeera, etc. They are all biased propaganda, just like the (NYTimes, Wash Post, Guardian, BBC, FT, WSJ etc), but you get a different biased perspective. I find that they are generally good at pointing each others lies.

> What news websites do you guys know/recommend to follow for news in these > countries? Language would preferably be in English.

Why not just follow vlogs of countries you are interested? Or visit the countries you are interested in? Or find some forums/irc/etc of the countries you are interested in and talk directly to the people?

News, by and large, is a great waste of time. Most of it is toxic nonsense written by people who know nothing about the subject they are covering.</comment>


High-Energy X-Ray Lasers Expose the Guts of Viruses from the Inside Out - <AUTHOR> http://motherboard.vice.com/read/high--energy-x-ray-lasers-expose-the-guts-of-viruses-from-the-inside-out

<comment><USER2> I'm pretty sure this is a continuation of research imaging the mimivirus at LCLS at SLAC.</comment>


The Megahertz Myth - <AUTHOR> http://www.youtube.com/watch?v=w9VWRB07yqc <comment><USER2> That video claims to have been uploaded in 2009, but it was made on 18 July 2001. Four years later, a funny thing happened to the PPC.

<http://en.wikipedia.org/wiki/Megahertz_myth>

[http://en.wikipedia.org/wiki/Apples_transition_to_Intel_proc...](http://en.wikipedia.org/wiki/Apples_transition_to_Intel_processors)</comment>


Dear-sir-or-madam: a bookmarklet that transforms web pages into ransom notes - <AUTHOR> https://github.com/notwaldorf/dear-sir-or-madam

<comment><USER2> Completely illegible due to the colors and lack of contrast. Would be greatly improved if newsprint gray + black were used.

<comment><USER3> I hate it when my ransom notes aren't legible.

</comment></comment><comment><USER4>
      javascript:var i,s,ss=['//raw2.github.com/notwaldorf/dear-sir-or-madam/master/ransom-it.js','//ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js'];for(i=0;i!=ss.length;i++){s=document.createElement('script');s.src=ss[i];document.body.appendChild(s);}void(0);

Do you _really_ need jquery for this?

</comment><comment><USER5> Substitute magazines instead of webpages and you've got an answer to one of the most famous interview questions.

</comment><comment><USER6> Perfect! I hope ransom notes come back in style.

<comment><AUTHOR> Yeah me too. For those who haven't checked it out yet, here's a screenshot of HN as a ransom note: [http://i.imgur.com/Io2bqof.png](http://i.imgur.com/Io2bqof.png)

</comment></comment><comment><USER7> May be it has some nostalgic/historic importance which I don't know, but if not, why would you want to do it?

<comment><USER8> It's sort of an old cartoonic reference to ransom notes which were assembled from cut out letters to hide the hand writing of the author. (obviously now u could just email them)</comment></comment>


Ask HN: Anything better than Tableau for data viz, dashboards? - <AUTHOR> I am in charge of implementing a web-accessible dashboard to visualise some data we are collecting on behalf of a client.
I am currently planning to use Tableau. However, my (admittedly very limited) exposure to the platform has left heavily underwhelmed. Tableau dashboards appear expensive, slow, ugly, and completely lacking in statistical tools.
Can anyone suggest a platform that can improve on any/all of these issues?
I am experienced with Postgres, JS, Ruby and Objective-C, and also with Stata and R, so I'm not afraid of some coding. But I am looking for something considerably quicker than coding the thing from scratch. <comment><USER2> As someone who has multiple years of experience maintaining tableau, it is laughably Ops unfriendly.

Want to change the email address that tableau sends reports to? Requires a restart of a Tableau.

Want to update the ssl certs used in tableau? That's a restart.

Want to upgrade tableau to a new veraion? Get ready to uninstall and reinstall the new version.

Of all of the servers and services that I manage, Tableau is my least favorite. However, apparently its incredibly good at what it does. I say apparently because I maintain it but I don't use it in day to day operations.

</comment><comment><USER3> If you want to leverage R, there is a web application framework called Shiny that lets you build interactive apps from R analyses: [http://shiny.rstudio.com/gallery/](http://shiny.rstudio.com/gallery/)

</comment><comment><USER4> If you're using more common visualizations (e.g. bar chart, line chart, scatter plot, etc) there's an excellent js library called C3 ([http://c3js.org/](http://c3js.org/)) that wraps charts implemented in D3 with a super-simple api. I'm a huge fan.

<comment><USER5> I did a lot of test plots with several D3-based charting libs, and found weirdness with C3, such as performance degradation over time, and oddities like including the label for the data as the first entry in the array. NVD3 seemed to be the most mature and sensible of all the libs I tried. [http://nvd3.org/](http://nvd3.org/)

</comment></comment><comment><USER6> I can't believe I'm going to say this, but tableau can easily be the right solution.

So, if you're dealing with large datasets stored in multiple systems (like Excel + MySQL + others) Tableau can be a boon.

The ability to use many different sources of data, create calculated fields that merge/modify other fields, and then operate against them? Quite nice.

I especially am happy with how I can create larger visualizations that work across different disparate datasets from many sources.

However, it does have quite a few issues in terms of UX, usability, etc. but so far I've liked it. Your mileage may vary :)

</comment><comment><USER7> If you want something with a lot of batteries included, extensive through JS/HTML5, and fast, look to QlikView [1]. Message me (address in profile) if you want someone to show you around the platform.

[1]: [http://www.qlik.com](http://www.qlik.com)

<comment><USER8> In my (admittedly limited) experience, you'll probably run into some of the same issues with QlikView being "expensive, ugly, and completely lacking in statistical tools".

<comment><USER7> I have deep experience in the QlikView (QV) platform, so I can address some of the points in your experience:

> expensive

QV is not free, that's for sure. You'll be spending tens of thousands of dollars for the one-time license fee, as well 20% yearly maintenance. On the other hand, you'll be saving thousands of hours of engineering effort by not reinventing the hundreds of wheels already in the platform. Don't succumb to "not invented here" syndrome.

Check out a few dashboards I designed in just a day total [1], or the vendors demos of Twitter data [2], HealthData.gov data [3], or Salesforce.com data [4].

> ugly

If you are a first time user, the default visualizations are ugly, no doubt. But for someone with design skill, the visualizations can be made quite beautiful. For wanting more tools, just add your favorite JS library and HTML to make a custom visualization.

> completely lacking in statistical tools

Fortunately, QV can link with R, allowing you to all the advanced capabilities you need. Need something more specific? Throw a microservice REST API on top of your desired application, and load that in through a GET request. There are Hadoop connectors also built in.

One thing that people often don't realize when comparing Tableau and QV, is that QV is a platform, as opposed to Tableau being just a visualization tool. QV includes ETL, task scheduling, and an in-memory analytics database.

[1]: [https://imgur.com/a/3Tzni](https://imgur.com/a/3Tzni)

[2]: [http://us-d.demo.qlik.com/detail.aspx?appName=Social%20Media...](http://us-d.demo.qlik.com/detail.aspx?appName=Social%20Media%20Buzz.qvw)

[3]: [http://us-d.demo.qlik.com/detail.aspx?appName=Epidemiology%2...](http://us-d.demo.qlik.com/detail.aspx?appName=Epidemiology%20-Tycho.qvw)

[4]: [http://us-d.demo.qlik.com/detail.aspx?appName=Salesforce.qvw](http://us-d.demo.qlik.com/detail.aspx?appName=Salesforce.qvw)

<comment><USER9> I have to agree with this assessment. Qlikview does ETL very well.

</comment></comment></comment></comment><comment><USER10> I like periscope.io myself, although I haven't done an in-depth comparison between the current options.

</comment><comment><USER11> You might take a look at Bokeh ([http://bokeh.pydata.org](http://bokeh.pydata.org)) and either the PyData stack or R (Bokeh can be used from R as well: [https://github.com/bokeh/rbokeh](https://github.com/bokeh/rbokeh)). Bokeh inside a Jupyter notebook with widgets and/or emerging "Bokeh Apps" is a powerful application stack. Anaconda is a single download that can help you get started with all the tools (including R): [http://continuum.io/downloads](http://continuum.io/downloads)

It still requires some coding but it is very powerful. There are a lot of examples in the Bokeh gallery and in examples directory: [https://github.com/bokeh/bokeh/tree/master/examples](https://github.com/bokeh/bokeh/tree/master/examples) .

There are several devs on Bokeh mailing list eager to help and the company behind Bokeh ([http://continuum.io](http://continuum.io)) can provide more significant help if you need it.

</comment><comment><USER12> I quite like BIME ([http://www.bimeanalytics.com/](http://www.bimeanalytics.com/))

</comment><comment><USER13> Disclaimer: I work at Tableau, but I don't claim to represent them, and I'm not in tech support or sales.

Two things that are likely: If you think the statistical tooling is limited you may not be aware that Tableau offers R integration built in. So if you're comfortable in R you can probably build what you want.

[https://www.tableau.com/new- features/r-integration](https://www.tableau.com/new-features/r-integration)

If you find it slow, it may be something that tech support can help out with (changing config settings, or reworking your dashboards to be more performant). You should email whoever manages your account, or hit up [https://www.tableau.com/support/request](https://www.tableau.com/support/request) and include your contact info.

You can also email me (email in profile) and I'll get back to you from my work account with the right contacts.

</comment><comment><USER14> I've used Dashing quite a bit, for relatively simple data, demo here: [http://dashingdemo.herokuapp.com/sample](http://dashingdemo.herokuapp.com/sample)

official page: [http://dashing.io/](http://dashing.io/)

</comment><comment><USER15> It really depends on what your use case is. There are still many unknowns related to implementing a web-accessible dashboard to make a decision.

What does your client intend to do with the dashboard? How much interactivity do they want in place? Is the data real-time? What sort of advanced statistical analysis does your client want to run?

Having answers to those might help guide you toward or away from Tableau. As an everyday user of Tableau with a solid technical background, there are some things that Tableau does well and there are some that it doesn't.

I love Tableau because it allows me to join many different data sources together quickly so I can analyze the data. I can easily drag-drop and visualize my data in many different dimensions. That said, sometimes the analysis is basic and it runs slower when there's a huge dataset.

</comment><comment><USER16> For Cyph, I initially looked into Tableau and various analytics platforms like Mixpanel, then ultimately realised that Google Analytics (which we were already using) had an events API that worked fine for our needs.

See: [https://developers.google.com/analytics/devguides/collection...](https://developers.google.com/analytics/devguides/collection/analyticsjs/events)

And this is what our dashboard looks like: [http://i.imgur.com/F8g8Hxq.png](http://i.imgur.com/F8g8Hxq.png)

It's fairly basic when it comes to visualisations, but thought I'd throw it out there in case it's helpful.

</comment><comment><USER17> I don't know about built in statistical tools beyond the basic aggregates available in SQL, but we (Fullscreen) have been using Chartio and are really enjoying it. I'd say it does great for 95% of the dashboards and visualizations we need (custom d3.js for the rest) and it plays well with our data sources. Particularly coming from Redshift data I found Chartio a lot snappier than equivalent charts in Tableau, especially for large data sets.

You can either use their UI to make charts, or write pure SQL, or my preferred method of making most of the functionality in the UI and tweaking the SQL for the last few details if you need something bespoke.

<comment><USER18> Chartio looks awesome, but I'm unable to get any pricing information from their site. Care to elaborate on the cost for a small shop to leverage this for a few dozen users?

</comment></comment><comment><USER19> Argo ([https://argo.io](https://argo.io)) is a web-based tool that enables fast, natural-language based question asking and visualization of data. It's designed to be used by a non-technical user, so you can share dashboards and visualizations with people, and they can ask their own questions.

Under the hood, Argo uses advanced search processors to turn natural language queries into SQL, optimized for visualization.

You can request a demo on their website: [https://argo.io](https://argo.io)

Disclaimer: I'm a co-founder and CTO of Argo

</comment><comment><USER20> Kibana / ElasticSearch? It's limited, but pretty and interactive, and gets you a bunch with very limited up-front work. I'm sure you can find some better demos, but here's one: [http://parlement.letemps.ch/](http://parlement.letemps.ch/)

HUE is a similar but different alternative. The "search" tab has some great demos, but appears to be down atm: [http://demo.gethue.com/](http://demo.gethue.com/)

</comment><comment><USER21> Though Tableau is incredibly expensive, I haven't yet found anything better that Tableau for web dashboard and visualization. Tableau is strictly a visualization tool and not statistical analysis tool. Tableau expects you to perform all the calculation on the back-end and send it the final data for visualization. I have also used Shiny, D3, Flot, Highchart, Chartio, Spotfire, Bime, FusionCharts,Qlickview and nothing comes close to Tableau.

</comment><comment><USER22> I'd suggest you use a JS library like HighCharts or D3JS (or both). All you need to do is format your JSON on the back-end in the correct format and throw it into the chart's configuration.

HighCharts has an amazing API, documentation, and examples.

[http://www.highcharts.com/](http://www.highcharts.com/)

[http://d3js.org/](http://d3js.org/)

<comment><USER23> What's your opinion on Highcharts usage longterm? I find it's great to get something up and running, but I've found myself hitting limitations, especially when it comes to custom design. But it's possible I may just be using it ineffectively. I always assumed that one day I should port my company's charts over to D3 if we wanted to be _serious_ about our data visualization (which makes up a big part of our site).

In other words, is it like Bootstrap in that it's a useful starting tool, but doesn't really scale well if you want to have full design control in the longrun? Do you happen to know any notable sites using Highcharts? Thanks in advance.

<comment><USER22> You can style almost anything on the chart with CSS and HTML I believe. Custom tooltip templates, etc

The API docs are just fabulous, with examples for everything :)

[http://api.highcharts.com/highcharts](http://api.highcharts.com/highcharts)

</comment></comment></comment><comment><USER24> QlikView is similar to Tableau, though more powerful and with a steeper learning curve. But if Tableau is too expensive, likely that QlikView is, too.

Another option is Looker, a relatively new product that relies more heavily on existing transaction/DW infrastructure. Dashboards are not ugly.

You can also look at d3, though by comparison development time will be much slower than the other two I've named.

<comment><USER9> If you've got SQL experience it's actually very easy to learn Qlikview scripting language. I recently finished a gig where the CEO retrenched all the IT staff and didn't bother to have any of the Qlikview dashboard processes updated. I had to pick up Qlikview in a few days, and had it all worked out pretty fully within 2 weeks. Learning "set-analysis" took me few days to get up to speed.

It's honestly not that hard to understand. The difficulty as always is putting together a sane data model.

</comment></comment><comment><USER25> You might want to try out Tibco Spotfire; it is similar to Tableau and Qlik.

They do provide some statistical tools (I personally don't know much about it). And they do have a WebPlayer to view the dashboards online.

See demos here: [http://spotfire.tibco.com/demos](http://spotfire.tibco.com/demos)

<comment><USER26> I use Spotfire regularly, and it's a pretty good tool. I highly suggest getting professional training, though, because our team didn't and it just took FOREVER for everyone to get in the swing of things.

It's really good for allowing your end users to explore data - it just requires a lot of development time to make really usable (ie: to make it more than "just a dashboard").

It has been built with R in mind from day 1. They have their own "Tibco Enterprise Runtime for R (TERR)" which I don't get to play around with much, but it's an obvious place to start for advanced predictive stuff, machine learning, and general data manipulation. Using the "RinR" package, you can pretty much do anything that R can.

My only absolute HATE with it is the LACK OF FREE SUPPORT/Community. Even though they have a nice "tibbr" (Facebook for businesses, basically) especially for Spotfire support, it's all behind a login wall, so it's not indexed by Google at all and it's not very searchable in my experience. In my opinion, this is a fatal mistake with their entire solution. Forums are amazing. Forum posts stay around forever. Very rarely do you want "the newest" forum post. You usually want a SPECIFIC forum post - making tibbr an awful user experience for support.

Additionally, their OLD forum/community IS indexed by Google, so you'll end up at dead-ends (404s with the exact information you want, conveniently highlighted in Google just before the answer is presented). Only a few people have blogged about it in the past, and even those are usually old versions. Also, basically no one talks about it on StackExchange. So, I just find it really hard to find answers to specific questions - like you might naturally do when programming to get a problem fixed quickly.

That said, it's super flexible and might be worth a look. I have very limited experience with Tableau and Power BI, but those lacked some of the convenience features I was used to when I used them. Personally, I wish I were forced to just program all my data visualizations in R or Python, haha.

</comment></comment><comment><USER27> VQL is a gui for really quick analysis and plotting. It's mostly for non- technical people, but we've had data science teams that use it to explore their data before breaking out IPython or Tableau.

I'm the founder of VQL. Nothing on our site yet, but happy to send a demo/instance. My email is jstrauss (then an @ sign) getvql.com

</comment><comment><USER28> One candidate is [http://redash.io/](http://redash.io/) My team has a backlog task to set it up. We use bigquery to hold onto our data, and redash can work with that.. It's open source, so it's free (except for one server) and looks good.

<comment><USER29> We use redash for some basic reporting. It is nice, and development is active.

</comment></comment><comment><USER30> My favourite right now is Looker. Dashboards are beautiful and ad hoc query creation is dead simple and on point. You will save yourself many "oh could you re-run this data but split it out by device type?" moments because your clients should be able to do it themselves extremely easily.

</comment><comment><USER31> Jasperserver with jasper reports has dashboards and is much improved now with visualize js. The dashboarding is only in the free version i think and not sure how the cost compares to tableau. It's much cheaper than business objects or oracle bi publisher etc though.

</comment><comment><USER32> [https://en.wikipedia.org/wiki/FusionCharts](https://en.wikipedia.org/wiki/FusionCharts)

EDIT: This seems to be heavily used in industry as well as in Federal IT dashboard...etc. So it appears to be a good choice.

</comment><comment><USER33> Interana ([http://www.interana.com](http://www.interana.com)) which is a YC S12 company sounds like your best bet.

Whats your scale? how many events per day/month etc?

</comment><comment><USER34> D3.js or even Google Charts will get the job done for most visualizations.

</comment><comment> bra-ket Pentaho, Saiku, or plain D3.js on the front end with your own middle layer

</comment><comment><USER35> A new product you could take a look at is [http://www.jut.io](http://www.jut.io). It's in beta / free for anyone.

It's a streaming analytics development environment, and uses d3 for visualization. It ingests both events and metrics.

It's based on a high-level dataflow processing language that allows you to process your data flexibly (moving window analytics, anomaly detection, general statistical processing). you can build interactive apps & dashboards and control which facets users can manipulate.

aaand here's the disclosure - I work at Jut and run customer success.

</comment><comment><USER36> Demoed Tableau at work a few years back and it was lacking in real-time visualizations. Is that still the case? If so, anyone have any recommendations?

</comment><comment><USER37> I'm a user. I haven't found something as good.

</comment><comment><USER38> Clicdata is user friendly and affordable. [http://www.clicdata.com/](http://www.clicdata.com/)

</comment><comment><USER39> Looker. Looker is the best. It's an upfront monetary investment, and the language you most need to know is SQL, but I love it dearly.

</comment><comment><USER40> [http://www.bimeanalytics.com/](http://www.bimeanalytics.com/)

</comment><comment><USER41> take a look at [https://event-fabric.com/](https://event-fabric.com/) we didn't officially launched the saas version but I can set up an account for you to try it for free of course.</comment>


Show HN: Cloudconnect, took me 1 yr to complete, need feedback to improve - <AUTHOR> https://github.com/sanketbajoria/cloudconnect <comment><AUTHOR> Powerful cloud-oriented client allow you to connect to various instances and applications via secure (SSH) tunnel & RDP

Features

Segregate instances under different profiles and workspaces for better management

Ability to connect with AWS

Connect to applications such as SSH, Scullog, Docker Machine, RDP & any custom Http/Https application

Create a forward and reverse tunnel

Share application safely with other, without sharing any connection detail of real instances

Secure workspace with strong encryption

</comment><comment><USER2> Looking through source code. Find that, there is no typescript support for it. I am planning to fork it, and provide type support.

<comment><AUTHOR> Good Idea. Please feel free to fork it. And, let me know, if you need any support from my side

</comment></comment><comment><USER3> Look promising.. will give it a try

</comment><comment><USER4> Awesome work</comment>


Reddit Forces Remote Workers to Live in San Francisco or Lose Job - <AUTHOR> http://sanfrancisco.cbslocal.com/2014/10/02/reddit-forcing-remote-workers-to-move-to-san-francisco-or-lose-job-tech-employee-fired-termination-relocate/

<comment><USER2> A note: these are being killed as dupes: [https://news.ycombinator.com/item?id=8404139](https://news.ycombinator.com/item?id=8404139)

Additionally, here's Yishan Wong's response: [http://www.quora.com/Is-Reddit- closing-their-NYC-and-Salt-La...](http://www.quora.com/Is-Reddit-closing- their-NYC-and-Salt-Lake-City-offices?share=1)</comment>


US Military Eyes 10 Megawatt Nuclear Reactors for Deployed Troops - <AUTHOR> https://www.defenseone.com/technology/2019/01/us-military-eyes-tiny-nuclear-reactors-deployed-troops/154406/ <comment> java-man putting fission material in the combat zone, what could go wrong?

we can't even account for nuclear material stored in the US, [0], [1]

who is pushing for this nonsense?

[0] [https://www.seattletimes.com/nation-world/nukes-on-the- loose...](https://www.seattletimes.com/nation-world/nukes-on-the-loose-how-it- happened/)

[1] [https://en.wikipedia.org/wiki/2007_United_States_Air_Force_n...](https://en.wikipedia.org/wiki/2007_United_States_Air_Force_nuclear_weapons_incident)</comment>


Sony's adorable new Aibo comes to the US in Sept, priced at $2,899 - <AUTHOR> https://techcrunch.com/2018/08/23/sonys-adorable-new-aibo-comes-to-the-us-in-sept-priced-at-2899/ <comment><USER2> I had a robotic dog as a kid. It was $20. You can adopt a shelter dog for $250-500. You can also dogsit for free.

I'm sure someone will pay nearly $3000 for this robotic dog, but I can't justify 1 month's salary on this when so many competitive options exist.</comment>


Silencing Many Hospital Alarms Leads To Better Health Care - <AUTHOR> http://www.npr.org/blogs/health/2014/01/24/265702152/silencing-many-hospital-alarms-leads-to-better-health-care

<comment><AUTHOR> My immediate though on hearing this story on NPR was "well, this is pretty much exactly the situation that affects much of system, site, and application monitoring".

For the engineer designing a system, providing an alert or alarm is the safe option. If something goes wrong and you _don 't_ alert on it, there's hell to pay. If something _doesn 't_ go wrong and you do ... well, CYA and all that.

The problem is that the person (or team) responsible for assessing alarms has to constantly filter signal from noise, and this can be difficult. Similarly for filtering out relevant from mundane information in system logs.

I've spent months re-engineering and tweaking Nagios and other monitoring such that noncritical situations _don 't_ trigger alerts, and properly configuring relationships such that fundamental errors suppress derivative ones. If you've configured Nagios to, say, send pages an alerts, you can rapidly DoS your own sysadmin / DevOps team when something _does_ go wrong simply from the cascade of alerts.

More specifically: following a few top-level health indicators, and _logging_ other metrics, has proven to be one of the most effective means of maintaining system stability. Literally a periodic load of a top-level (or representative) page(s), with response time indicated, and color-coding to indicate if you're in the healthy (green), warning (yellow), or unhealthy (red) zone, plus logging of results. It's the sort of thing that can be crafted in a small shell script and displayed in a corner of your monitor.

</comment><comment><USER2> This was problem on russian space stations as well. There was constant stream of alarms, and astronauts practically ignored them after 6 months.</comment>


Ask HN: Is the Blockchain Hype Ended? - <AUTHOR> <comment><USER2> I think this is an interesting question because HN is relatively quiet relative to the bigger events deep in crypto land (eg. new protocols / markets / etc), whereas for everything else tech, speculative or not, you usually see HN activity.

So I think the hype is still on its way up (looking at things like DeFi), but I think the overlap between the crowd that's hyped + HN readers is slowly decreasing (even as a lot of that crowd reads HN today).

</comment><comment><USER3> In my personal opinion, it is just getting started. There are people from many countries who are looking at crypto as a means to do their living and blockchain is getting more and more visibility. There are govts. trying to block the blockchain developments because of the traceability and immutability properties.

</comment><comment><USER4> [https://defipulse.com](https://defipulse.com) is the new hype. The ICO hype has turned into DeFi hype. Plus NFT's and decentralized storage hype around the corner.

Reddit also added ETH-based coins to some subreddits this year and is doing an ETH scaling competition - [https://www.reddit.com/r/ethereum/comments/hbjx25/the_great_...](https://www.reddit.com/r/ethereum/comments/hbjx25/the_great_reddit_scaling_bakeoff/)

[https://www.reddit.com/community-points](https://www.reddit.com/community- points)

</comment><comment><USER5> Not yet I think. I still see a lot of people trying to use BlockChain to start a company that could totally be done without blockchain and their primary aim is to use blockchain than solving the problem. Personally met 1 guy and see many on reddit etc.

</comment><comment><USER6> Based on the number of git activity that's publicly viewable (not to mention the actual on-chain transactions which are viewable across the hundreds of different blockchains), I'm going to say no.

</comment><comment><USER7> Definitely not over. A lot of investment going into the sector.</comment>


APL compiler targeting a typed array intermediate language - <AUTHOR> https://github.com/melsman/apltail/ <comment><USER2> This is awesome.

I always recommend that people at least look into APL, to get a taste of how different some paradigms of programming are from what they're used to. Unfortunately, the de factor implementation of APL (Dyalog) is proprietary, and last I checked, the process of getting a trial license takes too much time to be worth it for people with only a passing interest.

It's great to see another open source implementation.

Here's my favourite video of APL in action. It'll blow your mind: [https://www.youtube.com/watch?v=a9xAKttWgP4](https://www.youtube.com/watch?v=a9xAKttWgP4)

<comment><USER3> not to take away from this, which is quite cool but:

GnuAPL is coming along nicely, and of course libre.

Dyalog has a 32-bit windows (though it works on wine middling well) only unregistered version [http://dyalog.com/download- zone/unregistered.htm](http://dyalog.com/download-zone/unregistered.htm) that other than a splash screen works great. Also if you're a student: I sent them like a 2 sentence email(from a .edu domain) that basically said "hey I'm a student, I wanted to play around with APL for data analysis" and they got back to me with a student license in like a couple hours. I am wary of proprietary environments I'm not going to argue for them, but getting a hold of a working+legal copy of Dyalog is actually pretty painless

<comment><AUTHOR> > GnuAPL is coming along nicely, and of course libre.

From what I have been able to determine, most APL performance comes from hand- tuned implementations of the primitives, and of specially recognised idioms[0], for which the interpreter generates specialised code. This seems to imply that building an efficient APL interpreter requires a significant amount of manpower to implement all these cases, rather than being able to get away with having a few simple good ideas. Are you aware of any GNU APL vs Dyalog performance comparisons?

[0]: [http://aplwiki.com/FinnAplIdiomLibrary](http://aplwiki.com/FinnAplIdiomLibrary)

<comment><USER4> Your implication does not follow from your correct observation. APLs (including J and K) work from the simplest principles, and from then on it is just optimization.

K was implemented by Arthur Whitney, who implemented all the primitives (and their special cases) himself. At some point, if I am not mistaken, Niall Dalton rewrote some parts to be even faster - most primitives are easy to do efficiently (though a stunt programmer can definitely do them more efficiently), and most importantly - most patterns are too. E.g. "a@<a" means "sorted a". "*a@<a" and "&/a" both mean "minimum value in a" (each is a different recipe to compute it). It's not hard to implement, and beyond the primitives, 200 patterns cover a very significant percentage of use cases, and are definitely within the realm of possibility for a single person, let alone demigods like Arthur Whitney, Niall Dalton, Mike Pall, or a small team.

<comment><USER5> Silly question: Historically, did anyone ever attempt to create the APL primitives as separate utilities? The crazy idea that keeps recurring in my mind is that one could have a UNIX userland made of APL primitives. k's mmap approach, avoiding I/O, is preferable. But even if one used a named pipe or UNIX socket to do IPC, perhaps it could still be fast enough to be useful or fun. Feel free to dismiss this idea, but please kindly explain why it would not work.

<comment><USER4> As jxy noted, APL2 actually was used that way in the mainframe days (I actually had the pleasure to use it, and it WAS a pleasure compared to any other facility offered on that system).

However, it's hard to replicate _unix_ userland as K primitives - Unix pipes work well because every participant streams along implicitly; however, one of the subtle and often invisible truths of APL and K programming is that everything sort of assumes the length of every vector/list/axis/dictionary is accessible. It's not something that _cannot_ be removed, it's just something that AFAIK _has never_ been removed -- and the implications of removing it are likely to make APL or K less concise. Streaming in APL and K tends to be explicit.

K and APL are ridiculously useful, and once you've actually grokked the idea, it's hard to look at conventional programming languages (Java, C, C#, even Python) and not feel disgusted at the useless extreme verbosity and fragile building blocks. Some people use them as their shell, and have no issue delegating to "standard" unix userland. In fact, an older version of K IIRC had its "create directory" operation implemented as {system "mkdir ",quote x} or something like that. It might not look that way to an outsider, but extreme pragmatism is strong with the K community.

<comment><USER5> What if someone took c.c, k.h, linenoise.c and linenoise.h and made a tiny k4 REPL for UNIX with commandline history?

Would the k community appreciate this as useful, pragmatic solution?

I am a BSD /emul/linux, /bin/sh, vi-mode, ESC / user. I need searchable commandline history for q)k) and I really do not like rlwrap as a solution.

sources:

[http://kx.com/q/c/c/](http://kx.com/q/c/c/) [http://raw.githubusercontent.com/antirez/linenoise/master/li...](http://raw.githubusercontent.com/antirez/linenoise/master/linenoise.c) [http://raw.githubusercontent.com/antirez/linenoise/master/li...](http://raw.githubusercontent.com/antirez/linenoise/master/linenoise.h)

<comment><USER6> For q/k in particular, you can just write a wrapper in q/k. Nothing stops you from doing

        while[1;0N!. 0:0]

you can save the history in a table and do whatever you want.

<comment><USER5> "Nothing stops you..."

Except my lack of creativity using k at this point.

Thanks for this.

As a k noob, I wonder: is thinking of solutions using iteration and control structures a bad habit, at least until I have command of the rest of the language?

Currently I'm using recordio to save my sessions; crude, but it works well enough:

      recordio q 2>log

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> An interesting project was kicked around here / proggit the other day, using apltail as the middle of a pipeline transforming APL code into optimized GPU code (Futhark), instead of targeting C. It is a pretty good display of ML's suitability for this sort of task.

I've always been impressed (amazed?) at the performance APL and its descendants get in an interpreted environment -- I'd be interested to see performance comparisons between the output of apl-to-apltail-to-C / apl-to- apltail-to-futhark and an APL interpreter.

<comment><USER8> I believe this is the paper in question!

[https://futhark-lang.org/publications/fhpc16.pdf](https://futhark- lang.org/publications/fhpc16.pdf)

<comment><USER7> Oh wow, hard numbers, thanks! That's very impressive, but strangely misleading in the graph towards the bottom that appears to clock TAIL->Futhark->OpenCL and Futhark->OpenCL exactly the same in more than half the cases (if you look at the raw numbers if appears that the second approach blows the first away in most).

<comment><AUTHOR> That's because the speedup graph cuts away at a 100x speedup, but some of these (somewhat silly) benchmarks are compute-bound and hence gets these crazy 1000x speedups on the GPU.

</comment></comment></comment></comment><comment><USER9> Shashi Gowda presented on APL implemented in Julia at JuliaCon today: [https://gist.github.com/shashi/9ad9de91d1aa12f006c4](https://gist.github.com/shashi/9ad9de91d1aa12f006c4)

<comment><USER6> This is quite interesting, especially the llvm jit bits. Are there any performance figures?

<comment><USER10> It's just a shallow transformation to julia, so it's fast ;). Would be fun to compare with a production quality APL implementation (I hear they have some fancy optimizations that we don't yet have in julia).

</comment></comment></comment><comment><USER11> I used APL professionally for about 10 years. It absolutely changes the way you see and think about solving problems with computers.

Give it a shot: [http://www.nars2000.org/](http://www.nars2000.org/)

<comment><USER12> Relatedly, I _highly_ recommend anyone read Notation as a Tool of Thought¹ if they haven't yet.

If I may inquire, what did you work on with APL?

¹ [http://www.jsoftware.com/papers/tot.htm](http://www.jsoftware.com/papers/tot.htm)

<comment><USER11> A range of applications from inventory management for electronics manufacturing to DNA sequencing and robotics.

</comment></comment></comment><comment><USER13> If APL is your interest and you like Smalltalk, the take a look at F-Script. It is an interesting take on array programming.

</comment><comment><USER14> This is one of a few GPU capable compilers currently targeting APL as a source language. I am the implementer for another system, called Co-dfns, which is designed to integrate GPU compilation into Dyalog APL. A third system is the APEX compiler that targets SAC written by Bernecky. We get together at the ARRAY workshop and PLDI each year and usually go over and compare our systems to one another. Each system is bringing some unique features and capabilities to the table, and it will be interesting to see these systems evolve over time.

More information on Co-dfns can be found here:

[https://arcfide.github.io/Co-dfns/](https://arcfide.github.io/Co-dfns/)</comment>


YC Founders at Work Interview: Posterous - <AUTHOR> http://ycombinator.posterous.com/yc-founders-at-work-interview-posterous

<comment><USER2> jl: Is it possible to upload the founders at work series to itunes? I usually listen to podcasts on my drive to and from work on the iphone...I'd guess there are others like me

<comment><AUTHOR> I'll look into this. Thanks.

</comment></comment><comment><USER3> It's pronounced pah-ster-ous? I've been saying it wrong for awhile.

<comment><USER4> me too, I have always pronounced is: post..ear..ous

<comment><USER5> I've always associated the name with "posterity" (though the post/poster association is also apparent) and taken that as a cue for pronunciation.

<comment><USER6> From what I've heard, it started as "preposterous" in terms of it's preposterously easy to use.

<comment><USER7> I heard it as the effort required to blog was preposterous, pre-Posterous.</comment></comment></comment></comment></comment>


Google employees will rally in protest of alleged worker retaliation - <AUTHOR> https://techcrunch.com/2019/11/21/google-employees-will-rally-in-protest-of-alleged-worker-retaliation/ <comment><USER2> One of Google's core problems right now is that they've hired an entire class of employees that spend all day every day on protests and activism instead of actually doing any work.

It took awhile for management to realize what a bad mistake that was, and now they're stuck paying for it.</comment>