Dropbear SSH, a lightweight alternative to OpenSSH - <AUTHOR> https://www.librebyte.net/en/network/dropbear-ssh-a-lightweight-alternative-to-openssh/ <comment><USER4> I use an ad-free, open-source Android app called SimpleSSHD that implements a Dropbear SSH server. Being able to SSH into your phone and wirelessly perform an incremental rsync backup of all your photos and data is life-changing compared to the hell that is cables and the MTP protocol.

Thank you to all these projects for delivering me from the clutches of MTP, I am indebted.

<comment><USER5> A long time ago, when I tried something like that, I was stymied because the SSH server did not have permissions to write to any directory which the document readers etc. could see.

Is this sort of thing still a problem? How did you get around it?

<comment><USER6> I haven't used SimpleSSHD but I use Termux, which gives me a "normal" terminal (no root required).

Since Android 8 I've lost write access to general locations in the sd-card but I still have read access, as well as read+write on the internal storage.

It has an ssh server which can be nice if you want to edit some files on the phone remotely but that's not needed for rsync. I have a shortcut on the home screen which triggers a script that runs an rsync command for photos and other stuff I want to backup. Also use it to reverse-sync a folder from my NAS, want to send a file to the phone? But the file in that folder and run the script.

<comment><USER7> I think that you need to run the command "termux-setup-storage" to be able to r/w again:

[https://wiki.termux.com/wiki/Termux-setup- storage](https://wiki.termux.com/wiki/Termux-setup-storage)

<comment><USER6> Thanks, doesn't work though. I have write access to the private directory for termux on the sdcard but nothing else (which unfortunately breaks my usage, I'll report it to them any day to see if a fix is possible).

<comment><AUTHOR> This is the way Termux works so far, External storage: Storage on external SD cards. Each app has a private folder on the external SD card, and interchange between them needs to use a special API not yet available in Termux.

[https://wiki.termux.com/wiki/Internal_and_external_storage](https://wiki.termux.com/wiki/Internal_and_external_storage)

<comment><USER6> It worked fine in Android 7, when I updated to 8 my scripts started getting permission-denied errors on writes.

I don't see anything in that link that says that sdcard (outside private folder) is read-only.

The way I read "Each app has a private folder on the external SD card, and interchange between them needs to use a special API not yet available in Termux." is that I can't read another applications private folder, but I'm not trying to - I just want (write+delete) access to the publicly accessible parts of the sdcard (lots of other applications do this, and termux did in Android 7).

<comment><AUTHOR> Well I have Android 7 and termux works in the way you described at first: I have write access to the private directory for termux on the sdcard but nothing else... other applications can access every DIR in sdclard due implements SAF: [https://developer.android.com/guide/topics/providers/documen...](https://developer.android.com/guide/topics/providers/document- provider) and I think this what termux doc refers to.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> A word of caution: Many (most) IRC spambot detectors check if your connecting IP is also running a Dropbear SSHd service. This can cause you to be k-lined in some instances, and it's not immediately obvious to basically everyone why the anti-spambot bots are flagging your connection. Of course, this isn't Dropbear SSHd's fault. Just something you might want to keep in mind if you use both of these things.

<comment><USER9> It is because dropbear is very common in embedded systems. They are commonly riddled with vulnerabilities, so they are getting hacked almost as soon as they are publicly reachable. This is not because of dropbear, but because they are typically configured with weak credentials that are newer changed. I guess IRC servers see a lot of spam from such devices, so they just drop all systems which has dropbear.

<comment><USER10> It's probably also because of dropbear since embedded devices often run old versions and dropbear seemed to be vulnerable to severe vulnerabilities in the past:

[https://www.cvedetails.com/vulnerability- list/vendor_id-1580...](https://www.cvedetails.com/vulnerability- list/vendor_id-15806/year-2017/Dropbear-Ssh-Project.html)

</comment></comment></comment><comment><USER11> If anyone is interested in _other_ lightweight tools to complement a minimal embedded linux distro, check out Troglobit's GitHub repo: [https://github.com/troglobit](https://github.com/troglobit). He has a collection of tiny apps perfect for embedded systems, such as...

      - mdnsd (not in Busybox),
      - merecat httpd (much more full-featured than busybox  httpd)
      - inadyn dynamic DNS updater
      - finit (IMO much nicer than busybox's runsv)
      - <USER2>
      - <USER3>
      - ntpd (with ipv6 support!)

He has been super responsive to requests as well.

<comment><USER12> This is great, thanks for sharing!

</comment></comment><comment><USER13> Very common in embedded devices. This link isn't working at the moment, so here's a link to the actual project: [https://matt.ucc.asn.au/dropbear/dropbear.html](https://matt.ucc.asn.au/dropbear/dropbear.html)

<comment><USER14> I followed that literarary-clock kindle tutorial at the weekend, and I think dropbear was the SSH client the networking hack used. Makes sense if it's designed to be low-resource.

</comment></comment><comment><USER15> Dropbear works well for preboot LUKS unlocking with remote servers.

<comment><USER16> Currently, all my remote servers of any import use LUKS to encrypt the PVs. My /boot is a tiny unencrypted filesystem containing just the kernel, and an initrd, which prompts for my decryption key before booting. (afaict, the standard setup)

For remote servers, I reboot them and then have to use a serial console to type in the LUKS password.

Are you saying that with this, I could put an ssh server in the initrd (and I guess I'd have to make sure network was up as well), that I could log in to to provide my LUKS password???? Because that would be ... beautiful.

<comment><USER17> For Ubuntu or Debian the dropbear-initramfs package should handle most of it, it looks like [https://hamy.io/post/0005/remote-unlocking-of-luks- encrypted...](https://hamy.io/post/0005/remote-unlocking-of-luks-encrypted- root-in-ubuntu-debian/) is an alright run through.

Another approach is to use something like OpenWRT as a bootloader then pivot_root into the real distribution after unlocking it - not sure there are any good instructions online for that though. I'm using it on a Raspberry Pi colocated 14000km away for [https://dropbear.nl](https://dropbear.nl), it works pretty well. Kexec is great for remote kernel upgrades too.

<comment><USER16> You are rocking my world. :D

When I first started switching my VPSs to having full disk encryption, I think it was around lenny though it might have been squeeze. Anyway, me and another peer thought it would be good practice to, while we figured we'd never cover every possible surface, find a standard deployment for debian VMs where even though we have no physical access to the hosts, wherever possible minimized the ability of an employee at a hosting company accessing our precious, precious bits.

The memory hadn't come back when I wrote my first comment, but one of the ideas we had at the time _was_ shoving sshd inside the initrd! But we concluded it would be hard -- involving not only making a static build of sshd (which I did some eons ago when I had foolish opinions concerning /bin /usr/bin) but also probably trimming code away from it or adding executable compression, and modifying the initrd creation scripts....either way -- too much complexity.

So I went the route previously described. Now I learn that not only is there an ssh implementation which i can statically link into a tiny binary (which helps some other projects...), but someone went threw the trouble of making a modified initrd package with it!

Fantastic. Look for an email from me soon offering help on a specific project I noticed on your github...

I'm well aware of building my own scripts that use chroot/pivot_root tricks -- I personally like using them for making small boxes that run everything from ram and keep no persistent state.

But just out of random curiosity, what's the advantage of using OpenWRT?

<comment><USER17> > But just out of random curiosity, what's the advantage of using OpenWRT?

I can't remember the exact reason, maybe it was because then the "bootloader" is completely decoupled from the main OS which makes upgrading kernels etc easier. It was about 5 years ago I set it up.

I should add, all the Debian initramfs work has been contributed by various people over the years - full credit to people such as the Debian maintainers, currently Guilhem Moulin.

<comment><USER16> > should add, all the Debian initramfs work has been contributed by various > people over the years

Oh certainly, I would have assumed it was.

All I know for sure is that many moons ago I would have loved this feature, could have probably done it myself at great great great effort but didn't want to, and now, hey, here it is :) progress!!

As for decoupling and lowering complexity... <tiny voice> occasionally i miss LILO... </>

I've been paging through the code (dropbear) so far, very clear. Glad that there's TCP forwarding, as it opens the door to another possible solution in search of a problem. Namely, with USB over IP tunneled through dropbear, a user would have the ability to plug in a yubikey or some sort of challenge response device. ; )

Also, I sent you a note.

</comment></comment></comment></comment></comment></comment><comment><USER18> I believe Dropbear still has limited to no SFTP support but otherwise I've been very happy with it.

Dropbear is the default SSH server for DietPi[0], a lightweight image for Raspberry Pi and other (mainly single board) computers.

[0] [https://dietpi.com/](https://dietpi.com/)

</comment><comment><USER19> In a recent vulnerability assessment that I performed, I was surprised to find out that Cisco is using Dropbear on products such as UCS Managed C240M servers.

<comment><USER20> So there is SSHD on ios... :-P

</comment></comment><comment><USER21> Check out the security track record before using this. There have been occasional RCE's.

</comment><comment><USER22> Often used with ARM, why is it never used by default on X86/X64 distros? I would think that the "lightweight" alternative would be lean and mean, kind of like Nginx vs Apache.

<comment><USER23> Nginx won because it was faster, simpler and more easily extended, not because it was "lightweight" per se.

In comparison, dropbear doesn't really do anything that ssh doesn't, and lags in a bunch of esoteric features that "most" people don't use but that inevitably some people do. Who wants to use a distro where one's preferred ssh-agent feature or X11 forwarding inexplicably doesn't work?

Dropbear is small and builds cleanly everywhere, so it's what you pick if you're size constrained or just need "an ssh" for your embedded environment and don't want to bother integrating something larger. No one specifically wants it at the command line on their "Linux" system.

<comment><USER17> Dropbear author here.

It used to have one unique feature, but OpenSSH has copied it now[0] :)

       dbclient host1,host2,user@host3

to onion-TCP-forward through a few hosts.

[0] [https://manpages.debian.org/stretch/openssh- client/ssh.1.en....](https://manpages.debian.org/stretch/openssh- client/ssh.1.en.html#J_10)

<comment><USER24> Hi,

First of all, thank you for creating Dropbear SSH. I would love to try it. I am currently using OpenSSH with PAM (Google Authenticator) and Ed25519. Does Dropbear support both PAM and Ed25519?

<comment><USER17> PAM support is fairly rudimentary and only supports username/password. ed25519 isn't supported - a few people have wanted it I might add it at some point. I haven't seen a real reason to go with that over ecdsa.

<comment><USER25> The reason to use Ed25519 over ECDSA is that ECDSA can't be used unless you have a good CPRNG. Just ask Sony what happens if you reuse a nonce with (EC-)DSA.

<comment><USER17> True, that could be a good reason. Forgot to mention and can't edit in the previous comment, there's a PoC ed25519 implementation I need to look at merging.

</comment></comment></comment></comment></comment></comment></comment><comment><USER26> Does this have anything to do with [http://bearssl.org/](http://bearssl.org/) ?

<comment><USER27> No, Bear SSL is recent, DropBear has existed for a very long time.

</comment></comment><comment><USER28> I needed a ssh server with some tweaks a few years ago. I must say that the dropbear code was very neatly written, easy to read and easy to understand. It made me to choose dropbear instead of openssh for my tweaks.

I would have used dropbear on my main machine as well, but it doesn't seem to support ~/.ssh/config

</comment><comment><USER29> Archived: [http://archive.is/9lu2S](http://archive.is/9lu2S)

</comment><comment><USER30> I recall I've had trouble finding good documentation on the equivalent of OpenSSH features in Dropbear. Stuff like restricting the client IP or disabling port forwarding... lesser-used features like these. It's been a while though.

<comment><USER31> The general pattern used by most of these "lightweight" implementations of system software is granular compile-time options for every additional bit of functionality.

[https://busybox.net](https://busybox.net) is a good example of this.

</comment></comment><comment><USER32> Dropbear SSH is also shipped into OpenWrt. It works great, including ssh keys, useful for ssh/scp automation.

</comment><comment><USER33> I only miss one option in the Dropbead, the _UseDNS No_ equivalent from OpenSSH.

<comment><USER17> That should be the default in Dropbear (as a compile-time option)?

      #define DO_HOST_LOOKUP 0

</comment></comment><comment> S-E-P Used Dropbear on many occasion on both iOS and Android, very quality this one

</comment><comment><USER34> "resource limit reached" When publishing something on hn kills it.

</comment><comment><USER35> Dropbear has been created in 2002.

From the project ChangeLog:

```

[..]

0.28 - Sun Apr 6 2003

\- Initial public release

Development was started in October 2002

```

..why is it surfacing now on HN? O_o

<comment><USER36> I mean it's good to remind people of older and less-known projects, but Dropbear is quite common for embedded Linux applications.

OpenWRT uses it for example.

</comment></comment><comment><USER37> > Resource Limit Is Reached :\</comment>


Up or Out: Solving the IT Turnover Crisis - <AUTHOR> http://thedailywtf.com/articles/up-or-out-solving-the-it-turnover-crisis.aspx <comment><USER2> _Very_ interesting.

The major thing I'd add is ideally you can be big enough that moving sideways is an option, from one tasty problem domain to another. Google can of course do this.

Another, and it's also hard, is to have hard enough problems that people will stay because it keeps being challenging enough. That works in research, I'm not sure how often it'll work at the other end of things.</comment>


Now We Are Baking – Graphics - <AUTHOR> https://www.i-programmer.info/news/144-graphics-and-games/13326-now-we-are-baking-graphics.html <comment><AUTHOR> [https://m.youtube.com/watch?feature=emb_title&v=iBpolaB4DqA](https://m.youtube.com/watch?feature=emb_title&v=iBpolaB4DqA)</comment>


What it takes to build great machine learning products  - <AUTHOR> http://radar.oreilly.com/2012/04/great-machine-learning-products.html <comment><USER2> A great and insightful article. A common theme I've seen in practice is folks who have a deep understanding of ML often run straight to applying the most sophisticated algorithms possible on raw data. On the other hand, people who know a bit about ML but understand the domain better start by applying intuition to data cleansing and then follow up with simpler algorithms. Without fail the latter group ends up with better results.

<comment><USER3> I think there are essentially two "deep" understandings of ML prevalent today. The first is more common: the ability to do the calculus, algebra, and probability derivations required to design complex ML algorithms combined with the CS knowledge to find/design a good algorithm and the software design skill to actually implement it on real, "big" data.

No doubt this is a difficult position to master and those who perform well are able to tackle lots of mathematical and computational challenges. They also are model builders who (have tendency to) relentlessly seek complex models in order to solve complex problems.

The other, rarer side is the learning theorist who may or may not understand the model building, algorithmic, and computational tools but understands well the theories which allow us to have reasonable expectations that the tools of the first group will work at all. These guys have a funny story in that they were the old statisticians who got a major egg-on-the-face after proclaiming that essentially all of ML was impossible. Turns out the first group managed to redefine the problem slightly and make major headway (and money).

\---

The thing I want to bring to light however is that the second group knows the math that bounds the capacities of ML algorithms. This isn't easy. It's one thing to say you recognize that the curse of dimensionality exists, but it's another to have felt it's mathematical curves and to build an intuition for what forces are sufficient to cause disruption.

The more experience you have with the learning maths, the more likely you are, I feel, to apply very simple algorithms, to be scared of "little x's" (real data) enough to treat it with great care, and to attempt to explore the problem space with confidence for what steps will lead you to folly.

\---

It's a fine line between the two, though. Stray too far to the first group and you'll spend a month building an algorithm that does a millionth of a percentage point better than Fisher's LDA. Spend too much time in the second camp and you'll confidently state that no algorithm exists that does better than a millionth of a percentage point over Fisher LDA... and then lose purely by never trying.

<comment><USER4> Our Data Mining (different field but somewhat related) professor had this quote "All models are wrong, but some models are useful" on the first day of university.

You can build an extremely complicated model that is not useful, where a simpler one might suffice.

<comment><USER3> I've heard that line referred to as Box's Razor. It's definitely the right heuristic, but it's interesting to see that even if your model is _right_ , you're still in trouble if it's too complex. This is a sort of bias/variance tradeoff.

</comment></comment></comment></comment><comment><USER5> I agree that the big wins in machine learning/(weak)AI are probably going to come more from figuring out how to better apply existing models and algorithms to real problems rather than from improving the performance of the algorithms themselves.

That said one shouldn't underestimate the amount of commonality between problems that to some people may appear unrelated. For example this post talks about the gains in machine translation performance from including larger contexts. The same principle applies to many other sequence learning problems. For example you have a very similar issue with handwriting recognition where it is often not possible (even for a human) to determine the correct letter classification for a given handwritten character without seeing it within the context of the word.

</comment><comment><USER6> The article is light on details. imo there are two major things your team needs:

1) Programmers that have the needed math skills, or mathematicians with the needed coding skills

2) A distributed ML framework

Solving problem one is not easy but it's straightforward.

Solving problem two is harder. While there are a lot of open source machine learning projects, almost all of them seem to have a focus of being used by a person and not a program. Moreover very few do distributed processing except for mahout (<http://mahout.apache.org/>). Mahout is promising but the documentation is still thin, and I'm not sure if it's getting momentum in terms of mind share yet.

<comment><USER7> What kind of math skills? What would a programmer need to learn in order to work on ML stuff?

<comment><USER8> Aside from algebra to do log or exp division addition and multiplications, you have to be versed in statistics. Most ml problems are solved on statistical bases. Although many of the algorithms have been solved, you still need to grasp the statistics behind it which is a bit more involved than calculating the odds of a die

<comment><USER6> Yes, many people that I know working on ML do not remember statistics well enough. Some keep their ignorance while relying on mathematicians on-staff (who can't really code), while others either start buying college textbooks or go to night classes. There are too many people who don't understand the algorithms being used.

</comment></comment></comment></comment><comment><USER9> In case anyone clicks his link to Variational Methods and is confused to find an article on quantum mechanics, as inspirational and arguably related as it may be, I think he actually meant to link to: <http://en.wikipedia.org/wiki/Variational_Bayesian_methods>

</comment><comment><USER10> Right now NLP is mostly limited to niche applications, like e.g. sentiment analysis and clever products build around it. I actually think the reason is that both natural language processing and machine learning are still in their early days.

Imagine all the applications for consumer products if algorithms would be really able to actually understand language (as far as you can understand something if you are a computer program and not a sentient human being), e.g. for example if we were able to do _real_ text summarization.

I believe this is not only possible, but not as far away as people think. However, to reach that goal we need to let go of that idea that NLP is mostly about clever feature engineering, but instead start building algorithm that derive those features themselves. Part of the problem is how evaluation is setup in NLP. What the best algorithm is, is decided based on who gets the best performance on some dataset. This sounds all nice and objective, but you will always be able to get the best performance if you try enough combinations of features (overfitting the testset) [1]. These small improvements say little about real world performance.

For the NLP people among you this is an interesting paper that tries to do a lot of things different: <http://ronan.collobert.com/pub/matos/2008_nlp_icml.pdf>

This is the corresponding tutorial, which is quite entertaining as well: <http://videolectures.net/nips09_collobert_weston_dlnl/>

[1] I think, this is less true for machine translation, where there are more and bigger testsets and less feature engineering going on.

<comment><USER11> Careful with the Collobert ICML-2008 paper. It has a very negative reputation among NLP researchers who actually know the area, just for its setup/evaluation. If you're interested in the methods (which I think are interesting), that group's later work is much improved.

<comment><USER10> Thanks I will look into it.

</comment></comment></comment><comment><USER12> Very nice article Aria. You quickly mention Pegasos as a scalable alternative to SMO. I agree that this works well for linear models. But despite the claim that Pegasos can be trivially adapted to kernel models I have never seen any implementation of a kernel Pegasos and I don't understand how it's even possible. Have you used Pegasos-style algorithm to fit non linear models?

On the other hand there exist alternatives such as LaSVM that can effectively scale linearly to large datasets (but the optimizer works in dual representation as with SMO and not like Pegasos).

<comment><USER13> So...are you saying you need the dual formulation in order to allow a kernel model?

<comment><USER12> No actually it's not the case. But I don't know how Pegasos can be adapted to use kernels. If you take figure 1 of the paper [1] you will see that the gradient of the objective function is used to update a single weights vector `w` at each step of the projected stochastic gradient descent. In a kernel model, all the support vectors cannot be collapsed into a single weight vector `w`. You would need to handle the kernel expansion against the support vectors explicitly. But then how to select the support vectors out of all the samples from the dataset while keeping the algorithm online? The Pegasos paper does not say mention it.

[1] [http://eprints.pascal- network.org/archive/00004062/01/Shalev...](http://eprints.pascal- network.org/archive/00004062/01/ShalevSiSr07.pdf)

<comment><USER14> The set of support vectors is just the set of training examples that have non- zero alpha parameters. To implement the gradient update you just evaluate the support vector machine on the example (using the explicit kernel expansion) and then if the example has signed margin less than 1 you add y * eta to the corresponding alpha value.

The difficulty with Pegasos for non linear kernels is the support set quickly becomes very large and so evaluating the model becomes very slow. Note that since the alpha values are not constrained to be non-negative (unlike the standard dual algorithms) the alpha values don't ever get clipped to zero-- instead they just slowly converge to zero. It's still (I think) one of the fastest methods in terms of theoretical convergence guarantees but perhaps not as fast as LaSVM or something similar in practice.

However, there's been a more general trend in machine learning to use linear models with lots of features instead of kernel models, partially because of these sort scalability issues.

<comment><USER12> Thanks for the reply. I was told by twitter that this is similar to the kernel perceptron which I don't know well either. There is a good introduction with python code snippet here:

[http://www.mblondel.org/journal/2010/10/31/kernel- perceptron...](http://www.mblondel.org/journal/2010/10/31/kernel-perceptron- in-python/)

However it seems that you need to compute the kernel expansion on the full set of samples (or maybe just the accumulated past samples?): this does not sound very online to me...

<comment><USER14> It's true you need to compute the kernel dot product between every example you see and every example in the support set (every example that ever previously evaluated to signed margin < 1). Whether it's online depends on your definition of "online" . It's definitely not online in the sense of using memory independent of the number of examples, since you have to keep around the support set. I think there are results showing the support set grows linearly with the size of the training set under reasonable assumptions. However, it is online in the sense that it operates on a stream of data, computing predictions and updates for each example one-by-one. It's also online in the sense that its analysis is based on online learning theory (e.g. mistake / regret bounds). A lot of learning theory papers use "online" in the latter two senses, which is confusing if you expect the former.

</comment></comment></comment></comment></comment></comment><comment><USER15> It's a very exciting time. I'm incredibly excited to see what goes on here. I previously explored an online education start-up idea and I'm really looking forward to seeing Ng and Koller change the world. I'm also very exciting to see machine learning on the radar. For me one of the biggest challenges is often making AI intuitive. As machine learning becomes more mainstream it will be on people's design radar and that will make it less hard to turn great algorithms into great products.

<comment><USER16> Partially, although in my experience over the past 4 years doing this stuff 1 hour cleaning the input data gets you thrice the output of 1 hour tuning the algos. Some algorithms are more sensitive than others, but in general, garbage in, garbage out.

<comment><USER15> I think in many cases you're correct. My point wasn't about the performance of the AI algorithms themselves though. In my experience most of the problems where I want to use AI the algorithm itself performs adequately. Getting the interaction with the algorithm sensible for a non-technical user is hard. If AI becomes prevalent enough that UI/UX people start thinking about it, I suspect it will be much easier to solve that problem, which to me is the bigger business problem with AI.

</comment></comment></comment><comment><USER17> I think this is pretty accurate. Here is an example from my own thesis research: I'm using machine learning to tune an (underwater) communication link, i.e. decide what modulation / error coding algorithms/parameters will yield good data rates in a dynamic channel.

At first i tried using an off-the-self classifier to figure out which parameters will work well. That failed because by the time i had sampled a decent proportion of the possible parameter values, the channel would change (the number of possible combinations is of the order of a few millions).

Turned out that the real problem is not learning the performance of the available parameters, rather it lies in "learning how to learn": i.e. my ML system needs to adaptively search the space, by responding to the history of previous explorations and their outcomes. This kind of exploration would be effective only with an understanding of how the underlying modulation/coding algorithms work and interact with each other.

<comment><USER18> Indeed. From our own experience: we use pretty much off-the-shelf maximum entropy parameter estimators for parse disambiguation and fluency ranking. In the past ~10 years most of the gain has come from smart feature engineering by using linguistic insights, analyzing common classes of classification errors, etc. Beyond l1 or l2 regularization, the use of (even) more sophisticated machine learning algorithms/techniques have not yet given much, if any, improvement for these tasks in our system.

What did help in understanding models is the application of newer feature selection techniques that give a ranked list of features, such as grafting.

</comment></comment><comment><USER19> Reading this reminded me of this recent post by Chris Dixon, which is also a good read: [http://cdixon.org/2012/04/14/there-are-two-ways-to-make- larg...](http://cdixon.org/2012/04/14/there-are-two-ways-to-make-large- datasets-useful/)

</comment><comment><USER20> My for-profit company (Brighter Planet) often gets product ideas from our data scientists; it's exactly what Dr. Haghighi is talking about.

For example: trying to model environmental impact of Bill Gates's 66,000 sq ft house during a hackathon -> discovery that we need fuzzy set analysis (<https://github.com/seamusabshere/fuzzy_infer>) -> new, marketable capabilities in our hotel modelling product ([https://github.com/brighterplanet/lodging/blob/master/lib/lo...](https://github.com/brighterplanet/lodging/blob/master/lib/lodging/impact_model.rb)).

</comment><comment><USER8> I have enjoyed the author's other posts via his prismatic blog here. It's one of the most interestings blogs to follow with only a few posts so far. However, this article falls a bit short. It feels rushed out, which is understandable.

I think it would have been better if this was just the first part of a multi- article write up on ML. With this one being an intro and follow-ups on specific approaches.

</comment><comment><USER21> Probably try PCA (principal components analysis) to help select the most important features of data, first, before going further in modeling it.

</comment><comment><USER22> The article doesn't mention two important things (and instead focuses on being clever - the opposite of what machine learning stands for). First, the deep learning algorithms that automatically create features. Second, the importance of gathering lots of data, or generating it.

If you have to be really clever with feature engineering, then what's the point of even calling yourself a machine learning person.

<comment><USER12> I agree that deep-learning is an interesting approach to learn higher level features. However it's still a long way from being a universal solution: for instance deep learning won't help you solve the machine translation or multi- documents text summarization problems automagically: you still need to find good (hence often task dependent) representation for both your input data and the datastructure you are trying to learn a predictive model for.</comment></comment>


Responsive Web Considered Harmful - e-sushi https://medium.com/cool-code-pal/responsive-web-considered-harmful-f3a2f075e971#.qm6lgcyu2 <comment><AUTHOR> > It is said that it was not until 2001 when the first website to adapt to the > size of the browser, Audi.com, launched. It was called "responsive web > design" because it was designed to respond to the changing screen sizes > being introduced to the Web at that time.

All websites were responsive until we started doing things like using tables for layout.

<comment><USER2> Shit's responsive:

[http://motherfuckingwebsite.com/](http://motherfuckingwebsite.com/)</comment></comment>


Scaling node.js to 100k concurrent connections - <AUTHOR> http://blog.caustik.com/2012/04/08/scaling-node-js-to-100k-concurrent-connections/ <comment><USER2> Before everyone gets excited about these big numbers, I would like to remind you that even higher concurrency can be achieved with even lower CPU and memory usage using Erlang. These numbers are good for Node, but don't use this as evidence that Node is magical and much better at handling large numbers of connections than other systems.

<comment><USER3> People are excited by Node doing numbers like this because there is a massive active Javascript community - with hundreds of thousands of people using Javascript all day every day at work. 0.01% of these people would ever consider learning Erlang, and even if they did they would not be able to use it at work - ever. As with everything having better features means nothing if nobody adopts. I am not saying nobody uses Erlang and I am not saying people are not adopting it - but the number are just not comparable to the Javascript community . Lastly just because you know a but of Javascript I realise that this does not mean you can architect massive real time systems. But it is like WOW even people that play casually aspire to having the best kit or playing for a top guild.

<comment><USER4> >As with everything having better features means nothing if nobody adopts

Well, it means my product is going to be superior since I went the better, if less well known, architecture. It's not like erlang is a little unsupported side-project of a language, it's actually older then javascript if you count the time period before it was open-sourced, and only a few years younger if you don't, and is used extensively by many industries.

Also, just because javascript the language is more well-known doesn't mean javascript the server architecture is more well-known. I would argue it isn't; when people want a highly concurrent, solid server, erlang is always mentioned.

Lastly, erlang is a pretty easy language to learn. I had the basics down in a day, I had a prototype pubsub server that could handle 50k connections in two. The syntax is a bit strange, and honestly it does get in the way sometimes, but it's not hard.

<comment><USER5> You're missing the point of Node. You can construct a web rendering application and its AJAX parts in one codebase. You can move fastly code between server and client rendering. etc etc etc.

Probably Erlang is "better". BTW, Java & C are quite fast also. Java applications, well written, do scale. If they don't, they are folks out there specialized to make them scale.

Also, Erlang is probably easy to learn as language. But when you develop a web app, you have enough other skills to keep up with. Let's name CSS for one ;) The human brain is limited in its capacity to remember API and language specifics.

Also, more popular means more libraries, which makes the product in turn better. This is why so many folks turn to PHP. It's not elegant, but everything you need is already here.

Now I won't argue that you may have good reasons to use Erlang for yourself, may it be because you like the language structure, like to write libraries by yourself, or so on. But it doesn't make it "superior", foremost not as a platform.

<comment><USER6> So your argument is that web developers are too stupid to remember Erlang. Tell that to all those Django developers that have to juggle Python, HTML, CSS and Javascript! They must be superheros! Ruby on Rails developers must be as well!

<comment><USER5> I never told they are stupid. Rather, I think a simpler environment enables more productivity for the developer. Assembler is hard, some people master it in incredible ways. Does that mean that C is useless? No. Node goes the way of unifying the web stack around JavaScript, I find it a least interesting. The future will tell the rest.

</comment></comment></comment></comment></comment></comment><comment><USER7> Garbage collection is disabled. How is this then relevant to any real world usage?

<comment><USER8> He's not running with GC permanently disabled, he's only disabled the automatic GC because of the huge overhead required (claiming 1-second pauses every few seconds). He also mentions it's trivial to enable manual GC and run that via setInterval/setTimeout/what-have-you.

</comment></comment><comment><USER9> Isn't this really scaling the underlying C runtime to 100k connections?

</comment><comment><USER10> I use Node in production. The main thing I like about it is that looking at system usage graphs while number of users grow, only thing that is going UP is bandwidth ;)

I'd really like to see a story of someone really having 100k connected browsers. My online game currently peaks at about 1000 concurrent connections, and node process rarely lasts longer than 2 hours before it crashes. Of course, using a db like Redis to keep users sessions makes the problem almost invisible to users, as restart is instantaneous. I'm using socket.io, express, crypto module, etc.

I'd really like to see real figures for node process uptime from someone having 5000+ concurrent connections.

<comment><USER11> I'm using C# for my game Tribal Hero (www.tribalhero.com). It's still in early beta so I've only had 450 concurrent users . Our CPU usage and memory usage barely moved from 0 to 450 users. We're using socket selects and not even async sockets which would have even better performance. It's also backed by MySQL though we want to eventually move to Redis. Why is Node breaking at 1k connections? Doesn't seem like much at all.

<comment><USER10> I also use MySQL as backlog, it's practically write-only as I keep the whole state in javascript objects. Only time when data is read from MySQL is at program startup. However, having SQL database enables me to run various complex SQL queries for reporting.

However, I do use Redis for one thing: user sessions. I turned persistence off as Redis seems to be rock-stable, and I really don't need sessions to persist. I was using a modified version of Node's MemoryStore, to which I added clean garbage collection, but with often restarts I mentioned earlier it has become pain for users to have to login again when in the middle of the game. Having a separate, dedicated Redis instance to handle the sessions made restarts completely seamless, as the cookie sent by user's browser remains valid between node restarts.

I was not willing to learn new db technology, but there wasn't really much to learn with Redis. You can set it up in minutes and it just works(tm). I highly recommend you try it.

<comment><USER11> I'll be switching over the entire game state to Redis. It'll be a bit of work but what I like the most is that it maps more naturally to objects. My db is mainly writes as well.

</comment></comment></comment></comment><comment><USER12> Can uwsgi/nginx be configured similarly?

Is it common practise to have node face the web without nginx?

</comment><comment><USER13> Link to his next post showing him breaking 250k - [http://blog.caustik.com/2012/04/10/node- js-w250k-concurrent-...](http://blog.caustik.com/2012/04/10/node- js-w250k-concurrent-connections/)

</comment><comment><USER14> It's a shame, that he didn't mentioned about kernel tuning. Without custom settings ( like net.ipv4.tcp_mem ), i think, it's a very difficult to reach this numbers.

</comment><comment><USER15> I did 3M/node on physical severs, 800K/node on EC2 instances.

We mostly use Erlang on server-side and node.js + CoffeScript on client-side (where they rightfully belong ;)

</comment><comment><USER16> It struck me the author runs his apps as root (in screenshots). But then I remembered he's using node.js to handle "thousands of concurrent connections".

<comment><USER17> I think it's his testing machine, so it's shoot & forget setup.

</comment></comment><comment><USER18> Looks like author is not aware of some cuncurrency problems, deadlocks etc. Backend/database might not scale to 100k concurrent connections so easily.

<comment><USER19> This is where NodeJS _really_ starts to shine - persistant connections and background operations let you do a whole bunch of cool stuff to mitigate that.

In my case I have entire db tables and collections replicated in memory and kept in sync via redis pubsub, and the 100,000s of concurrent users I have are all sharing just a few dozen persistant redis and mongodb connections between them.

</comment></comment><comment><USER6> I would really love to know what he did to tune that Rackspace VM. I had a terrible time trying to get node.js and others to get past 5,000 concurrent websocket connections on a m1.large EC2 instance or on Rackspace.

</comment><comment><USER20> I wonder what happens at 100k database connections , i will give a try with firebird and the nodejs driver

<comment><USER21> That's the thing about these types of benchmarks. They're useful for showing that node has the throughput -- at a low level -- to serve a huge number of concurrent connections, but it doesn't translate directly to huge application throughput if you're relying on things like database access over a network. In practice, each of these problems must be solved individually.

I don't mean to minimize this accomplishment. If you're assuming you need 100k database connections in order to scale, you might be solving the wrong problem. Scaling is a matter of moving data as close to the CPU as possible. This means in-memory caching is where real performance comes in. I don't care how good your language/framework is, you can't defeat the physics of slow I/O over a network.

</comment></comment><comment><USER22> i remember seeing this on HN back in April</comment>


Nethack: The Greatest Game You Will Ever Play - <AUTHOR> http://thegreatestgameyouwilleverplay.com/

<comment><USER2> Having beat Nethack five times I can't say it's not a _great_ game. That said, there are a lot better games, and even many better roguelikes. Dungeon Crawl Stone Soup is my recommendation. It's insanely difficult, but in a better way than Nethack (you get killed when you make tactical mistakes, not when you trip down stairways, get crushed by drawbridges, run into various insta-kill monsters, etc). Not that I want to discourage anyone from trying Nethack... it's just not the best introduction to the roguelike genre.

</comment><comment><USER3> Surprisingly stylish site given the game itself.

</comment><comment><USER4> Is this a joke or not? I'm sorry, I can't tell.

It's not that the game is bad, it's just that it won't appeal to the same people the website design appeals, for example.

And it's a very strange game.

<comment><USER5> Who does that website design appeal to? I found it hard to read. It looks like a one-page spread in a magazine that I'd have skipped over.

<comment><USER4> People who like fancy graphics and typography?

</comment></comment></comment><comment><USER6> Wow, only 35k users?

</comment><comment><USER7> I used to think 'Nethack' was multiplayer over the Internet - the 'Net' bit - but it doesn't appear it is. Is it?

<comment><USER8> No, there's limited interaction by means of 'bones' files if you play on the same host.

If you die and certain conditions are met, you'll leave your corpse, all your items, and a ghost for another lucky (or unlucky, given that whatever killed you will still be around) adventurer.

There do exist some multiplayer roguelikes, Mangband <http://www.mangband.org/> is probably one of the most well-known/popular.

</comment></comment><comment><USER9> I know it's kind of a spiritual successor, but Dwarf Fortress might just take the cake...

</comment><comment><USER10> I'm suddenly tempted to give SLASH'EM another go.</comment>


Snowden twist: disrupting secret Swiss banking cartel agreement. - <AUTHOR> http://reason.com/blog/2013/06/10/switzerland-furious-about-snowdens-charg

<comment><USER2> On the bright side, now the extremely wealthy will be throwing money at anti- snooping efforts. They may have poked the wrong bear.</comment>