Ask HN: Employer Violated NDA - <AUTHOR> I just recently left a company and on leaving I signed a NDA basically saying that neither party should discuss the reason for the split to anyone. So, on the same day I left, the founder had a meeting with the other employees and told them the exact reason I left thus violating the NDA.
We have meetings frequently, so it's customary for them to be recorded. One of my former co-workers sent me audio of the meeting and he violated the NDA to the fullest extent.
I'm really not familiar with the legalities of this situation. I wanted to know what my options are, and if anyone has any recommendations. <comment><USER2> There are two types of NDA: Mutual and Unilateral.

If it is mutual, you likely can take action.

If it is unilateral, you likely don't have a case.

You can tell by the wording and possibly the title of the NDA.

In any case, lawyer up if you want justice. There may be something said on the recording that could be slanderous, libelous, etc. And then of course, it may not be admissible in court if it was recorded without the speaker's knowledge. Or, perhaps someone who was there would testify in a deposition that it was in fact said. Repeating myself, lots of good questions for an attorney...

<comment><AUTHOR> Yeah. It was mutual and slanderous.

</comment></comment><comment><USER3> NDAs are typically not worth the paper they are written on, but consult a lawyer if you think it could help.

Fact of the matter is, the damage is done, there is no taking back what he said or did. Moving forward your choices are to let it go(fuck those guys anyway right!), or fight for something you probably won't get.

The process of getting lawyers and pursuing this in court is not an easy process, it will wear on you mentally, it will make you want to rip your fucking hair out in frustration and, in the end, probably nothing will come out of it.

</comment><comment><USER4> If you signed an NDA with the company (not the founder personally) then it is probable that spreading the information within the company is fine.

</comment><comment><USER5> These agreements typically restrict the employer from speaking to outside parties, not other employees. Re-read your document carefully — there's almost always a carve out for disclosing to others who are bound by a confidentiality agreement (as employees would be).

In short, you almost definitely have no recourse, but check with an employment lawyer to confirm that.

</comment><comment><USER6> When you say "neither party", you're talking about 2 parties. Was the other party the company? Or was it with the founder? This is an important distinction.

This is why _seemingly simple_ things are defined explicitly in contracts.

</comment><comment><USER7> Consult a lawyer, not HN. HN is the wrong place to go for the legalities on any situation.

<comment><AUTHOR> What type of lawyer?

<comment><USER3> See my other comment for my opinion, but if you want a lawyer, Maria Kallmeyer at Quarles in Chicago is a bad ass, she might be able to advise you or help in some way.

email@example.com

<comment><AUTHOR> Thanks.

</comment></comment></comment></comment><comment><USER8> Lawyer up.</comment>


Tim Cook, privacy martyr? - <AUTHOR> http://www.economist.com/news/business-and-finance/21693189-apples-boss-may-have-choose-between-his-principles-and-his-liberty-tim-cook-privacy?fsrc=permar|image2 <comment><AUTHOR> Will Tim Cook be held in contempt of court and go to jail? What are your predictions?

<comment><USER2> I predict the government will go after Apples billions of overseas dollars for leverage.</comment></comment>


An opinionated view of the Tidyverse "dialect" of the R language - <AUTHOR> https://github.com/matloff/TidyverseSkeptic <comment><USER3> From personal experience, I learned R during college as a part of my stats courses, before tidyverse even existed.

After doing a few personal projects using base R with complex manipulations, I would have _quit using R entirely_ if it were not for tidyverse/dplyr.

I'm a _pragmatic_ software engineer: I prefer to get a data analysis job as fast as necessary, and IMO there isn't much merit in making things more complicated by using base R just for the sake of base R. It's definitely not _faster_ , both in cognitive load and in speed. In my data science work there hasn't been a _single_ case where I've needed to fall back to base R, or wanted to.

Pipes are being undersold here: the examples only show a single piped action, whereas pipes shine when you have to do 5+ operations in a single manipulation (I've seen base R code that does that by just having a _lot_ of redundant df assignments and it's a pain to read).

<comment><USER4> With regards to the usefulness of piping with dplyr: the post does express its admiration for data.table as the alternative for dplyr.

In data.table, chaining operations together is standard and as simple as a set of brackets to contain the next operation.

Both are preferable to pandas though (joke not troll).

<comment><USER5> There are nuances that to me make R data frames, especially with Magrittr, simpler to work with over Pandas. Pandas indexing always seems to get in the way.

I have been using Python a lot longer than R, but the Pandas syntax is taking longer to internalize.

Maybe it's just me?

<comment><USER6> No. Pandas is a powerful tool, but a poor api / user interface.

<comment><USER7> Pandas grouping and slicing is, imo, significantly worse than R. And I so despise those multiindexes.

</comment></comment></comment></comment></comment><comment><USER8> He is glossing over the major issue with base R, which is its tendancy to switch data types in a way that appears random to new users. As an undergrad I spent nights literally on the verge of tears debugging R code where the types had been mucked up by R's bizarre semantics. Claiming that data[foo] is the same as filter(data, foo) is not correct - the [] and [[]] operators have a lot of strange side effects depending on what argument is passed in. It is like saying map() and a for loop are the same because everything a map can do a for loop can do. That is true, but the single-mindedness of map is a major selling point because you know it is only going to do certain specific things. The issues here will disproportionately affect beginners who are likely to accidentally pass in the wrong thing - it needs to fail with an error or in a predictable way to best help them. Hadley's code fails sensibly.

One of the _major_ innovations of the tibble package is that it puts the types of each column into the headings when it prints the table out. Dr. Matloff is grossly underestimating how painful it is for beginners to, eg, figure out when they are dealing with a column of strings vs a column of factors. If a beginner copies code to someone else for help, the other person can actually tell what types are involved! It is like magic compared to base R.

Given how ad-hoc the R community is and the paucity of people with programmer backgrounds, these minor usability improvements on controlling types are important. And anybody who cares about efficiency is using the wrong language.

 _EDIT_ Also, this comment that RStudio is bypassing an Open Source project's leadership - that is one fo the major features of open source. It is usually a healthy sign that the project leadership are lost in the weeds. I recall a situation with broad parallels in the history of GCC for example; it didn't kill the project although I believe a bad codebase died.

<comment><USER9> > "Dr. Matloff is grossly underestimating how painful it is for beginners to, > eg, figure out when they are dealing with a column of strings vs a column of > factors."

This is just sapply(df, class)... Is this not found in R 101 material that beginners are likely to find?

<comment><USER10> Pay no attention to the myriad of other apply functions hiding behind the curtain and the difficulty in remembering what each of them does and why you can't just use map, eh? :)

So to answer your question, no, its not just simply that.

<comment><USER11> I use a simply device to keep these straight:

\- (l)apply: List apply always returns a list

\- (s)apply: simplify apply tries to return simplified result

\- (v)apply: verify apply checks the return type conforms to user supplied example

\- (m)apply: multiple apply applies FUN to multiple vectors

\- (r)apply: recursive apply is essentially a flatmap

\- apply : no device here, only use on matrices, never data.frames

<comment><USER8> After reading your explanation I still, as has been the case for years, don't understand what sapply or rapply does, and vapply sounds weird.

That isn't going to change, because I'm just not going to use them or 'invest' the time in finding out what some statistician-of-yore's interpretation of a map is. Instead I'll stick to tidyverse map - returns a list. Or tidyverse map_[int/chr/dbl/etc, etc] if I want a vector of [int/chr/dbl/etc, etc].

That and the data frame manipulation verbs covers the most useful 80% of cases where *apply would otherwise be needed. If the base R team were implementing functions that way in the base, stats professors wouldn't need to complain about mass exoduses from base R.

<comment><USER11> Yeah, that's a valid point. And it took me a couple years to internalize these differences -- mostly by being burned on numerous occasions. Base R has a lot of idiosyncrasies and they have to be memorized, unfortunately.

I develop R packages for my colleagues and so I stick to Base R whenever possible. I don't want my packages depending on the tidyverse at all. But for EDA, I am agnostic about what my colleagues do. They should use the tools that stay out of the way and let them get their hands around the dataset intuitively. For me that's Base R, for others it's data.table or tidyverse.

</comment></comment></comment></comment></comment></comment><comment><USER12> In this article and others I've read, people complain about the tidyverse's lack of performance, but I think that places too much emphasis on speed of execution versus speed of development. As an academic, most of the R users I know only code as a portion of their scientific projects. Besides data analysis we're doing data collection, manuscript writing and grant writing. The tidyverse's more english-like syntax (eg select() versus `[`) and following a series of pipes rather than unnesting ten sets of brackets makes it so much easier to come back to my code after a few weeks or months and pick up where I left off, or to work with other people's code. My time is more valuable than computer time so the tidyverse is my choice.

<comment><USER13> This is my experience as a data engineer/analyst. I work in the healthcare space and my analyses are run on datasets that are at the uppermost half a million rows. Yes, data.table is faster than dplyr/tibbles, but i dont care. Like you said, most of my time is spent on expressing my thoughts into code, not running the code. Tidyverse really simplifies it compard to base R.

Its good to know tools like data.table exist though, people shouldnt think the tidyverse is the only way to do things.

</comment></comment><comment><USER14> As an R user of some 10+ years, I think some of this is spot on and some of it is just silly.

Pretty much everyone knows the Tidyverse is slow to run, but many of us find it faster to write and very readable because it avoids one-time-use assignments (i.e. in `a <\- f(x); g(a)` the variable `a` is only used once). Naming things is hard. Prof. Matloff's point is well taken though - you don't _need_ the Tidyverse and it doesn't run faster. Fair criticism.

> The "star" of the Tidyverse, dplyr, consists of 263 functions. While a user > initially need not use more than a small fraction of them, the high > complexity is clear. Every time a user needs some variant of an operation, > she must sift through those hundreds of functions for one suited to her > current need.

> By contrast, if she knows base-R (not difficult), she can handle any > situation.

Yet base R has some 1,346 functions. Another 455 functions in `stats`. I'm not sure the quantity of functions has as much bearing on learning as the quality and consistency. So this seems like a silly argument to me.

Mostly I think Prof. Matloff is forgetting that base R is kinda terrible and no one else has done much about it aside from RStudio.

<comment><USER15> > Mostly I think Prof. Matloff is forgetting that base R is kinda terrible and > no one else has done much about it aside from RStudio.

I think he'd disagree with the contention that base R is kinda terrible. Like, I'm pretty sure he's wrong. But this isn't something he's forgotten, it's something he's never believed.

The point about the Tidyverse and RStudio is that RStudio doesn't make any of its revenue from the Tidyverse. They distribute it free. There are no upsells for any of it -- I can't go out and buy ggplot2 add ons, I can't buy dplyr support contracts. There's no money to be made on the Tidyverse, and plenty of resources thrown at it.

So what is RStudio selling? They sell an IDE, they sell servers to host things built in R, they sell a package manager solution, they offer most of these things as a SaaS offering as well (although rstudio.cloud is still in trial and they haven't started charging yet).

In other words, RStudio offers tooling for people who write R. They don't really care what kind of R you're writing, from a financial perspective. They get paid the same if you're using RStudio to write stuff using data.table as they get paid if you use dplyr. RStudio is making a bet that more people will want to write R code if they make R a more pleasant language to work in. And anecdotally, they're absolutely right -- I tried to learn R before the Tidyverse, and I hated every moment of it. I like post-Tidyverse R (although I got into it when people still called it the Hadleyverse). I know several other people with similar experiences to mine.

<comment><USER16> If you don't think base R is kinda terrible then I recommend you read R Inferno. It's truly a crazy language with enough footguns for a centipede. A lot of the attraction to the tidyverse is having sane defaults and consistent APIs that help you do the right thing quickly.

</comment></comment></comment><comment><USER17> Decent critiques in my opinion. I was trained on/learned base-R myself ~10 years ago and love the "tidyverse" (although the author is correct, data.table is superior for big data and its not particularly close). Can't imagine people that only know tidyr/dplyr/etc without knowing base-R, seems like those people would get exposed quickly.

I enjoy base-R solutions where I can. I use data.table when my data sets are medium-ism. But like most people, my data is small and people just call it big and thus the attraction to dplyr in particular. Those chained pipes are addicting.

Always love seeing R discussion on HN, seems to be taboo though (not a 'real' language and all that).

<comment><USER18> I don't even understand the not a real language critique, R:

\- is based on scheme, a HN favourite;

\- integrates very well with C++ through RCpp allowing you to do whatever you want. 9/10 times someone already went through the trouble for you.

<comment><USER19> The R standard library is kind of weird and hard to use for "general purpose" programming. It's very clearly a domain specific language.

Also, it's unbelievably slow for basic operations like looping, function calls, and variable assignment. It's literally orders of magnitude slower than Python (I've tested it). Unlike its fellow C-flavored-Lisp Javascript, R retains an extreme level of homoiconicity, Which apparently makes the language very difficult to compile or optimize. I don't know how e.g. SBCL does it, but over the lifetime of the R language no one has managed to implement a non- trivial subset of R that performs better than GNU R (fastR was never finished to my knowledge). So you are basically relegated to writing code in C, Fortran, or C++ if you need even decent performance. Otherwise you are stuck using "vectorized" operations like lapply(), which are fine but make for a jarring experience if you're coming from other languages.

So of course it's a "real" language in the sense that Bash is a real language. But it's ultimately and fundamentally a domain-specific scripting language and I don't know if there is a way around that.

<comment> vincent-toups I can give you some sense of why one can compile SBCL and not R. SBCL and other Lisps these days maintain some kind of conceptual barrier between macroexpansion time and run time, so that you can _stage_ your activity appropriately. Read the code, gather the macro definitions, expand the code (repeat as necessary) until you have some base language without metaprogramming in it.

Then you can optimize and compile. This is particularly true of Scheme which foregoes much of the dynamic quality of Common Lisp in favor of a much more static view of the world. But its still basically true in CL.

R doesn't have macros of that kind. It has a sort of weird laziness based on arguments remaining unevaluated until needed. Metaprogramming in R is typically done by intercepting those "quoted" arguments and then evaluating them in modified contexts (this is exposed to the user more or less by letting them insert scopes into the "stack").

Thus, there is no distinction between macroexpansion and execution time. Hence, its tough to write a compiler, which is basically just a program which identifies invariants ahead of time and pre-computes them (eg, lexical scope is so good for compilers because you can pre-compute variable access). Because all sorts of shenanigans can be got up to by walking quoted expressions and evaluating them in modified contexts, R is hard to compile.

This is, by the way, why the `with` keyword was removed from Javascript. It provided exactly the ability to insert a scope into the stack used to look up variable bindings.

<comment><USER20> > This is particularly true of Scheme which foregoes much of the dynamic > quality of Common Lisp

The idea of even the standard Common Lisp is that both is possible: a static Common Lisp and a dynamic Common Lisp, even within the same application in different sections of the program. Common Lisp allows hints to the compiler to remove various features (like fully generic code being reduced to type specific code), it allows a compiler to do various optimizations (inlining, getting rid of late-binding, ...) - while at the same time other parts of the code might be interpreted on the s-expression level.

<comment> vincent-toups Its definitely the idea. I know it works well enough for a lot of people, but for me, I'm happy to just give up on the dynamic behavior in favor of a simpler universe.

The last thing I want to be thinking about is whether my compiler needs a "hint" that something can be stack allocated, for instance. That is their business, not mine.

<comment><USER20> Common Lisp as a language standard makes no commitment to the smartness of a compiler. If a certain compiler can figure out things, great, but some compilers might be dumb by design (for example to be fast for interactive use). That's left to implementations.

</comment></comment></comment></comment></comment></comment></comment><comment><USER21> I think he's exaggerating the effect of some of these claims. People who "grew up" learning R+Tidyverse will discriminate against non-Tidyverse users? Give me a break. You are either an advanced or competent or novice R user. You either understand how pipes (%>%) work or you don't.

I doubt that anyone is going to be denied a job because they are an _amazing_ R programmer but they just don't have the experience with a particular set of packages, especially those that are as well implemented and easy to learn as the Tidyverse.

I can totally imagine someone not getting a job because they are a crap programmer, and then blaming it on something else. Or I can also imagine someone saying that the Tidyverse packages suck, and then be denied a job because of their attitude.

I could make a similar argument for most of the substantive claims in his post.

Some of these examples (dplyr vs. data.table) are cherry picked. I have several of my own examples where read_csv is way faster than read.csv, so maybe, like all good programmers, we should be testing and profiling our code and implementing the parts that make the most sense for our needs.

The bottom line is that the Tidyverse is a good set of packages and you are free to use them or not. There isn't some blood feud (like vi vs. emacs) between users and non-users, we all get along just fine.

There are dozens of great tutorials on how to learn R that don't use the Tidyverse, and RStudio is under no obligation to offer a full course on every possible way to learn R. Any R user is also a competent Google user.

It's fine to be opinionated. But, a professor as respected as Norm Matloff should be careful of how they say things, or they risk souring their students on a set of packages that might be very useful in the future.

<comment><USER22> "cherry picked"? There is no doubt that data.table is generally way faster than dplyr, unless you cherry pick a few use cases.

The problem is that dplyr is much slower than data.table and RStudio is promoting tidyverse too much to make the slower choice a default for many users.

The article is 100% correct on this issue.

<comment><USER15> Depending on the size of your data, you might not care that dplyr is slower than data.table. If you're better at writing/composing dplyr, you can often make up the speed difference between the two in terms of the savings in time spent writing and reading code. And if your data is that large, there are solutions like dbplyr out there to run dplyr code on various backends and offload the computation outside of R.

<comment><USER3> In practice, if there's ever a case that there's "too much" data such that dplyr starts to hang (e.g. millions of rows, hundreds of columns), you would get better value by setting up a database first with the data. Which you can then query with dbplyr!

<comment><USER22> data.table can handle millions of rows easily, as long as the data can fit in the memory.

</comment></comment></comment></comment></comment><comment><USER23> As someone who uses and teaches R extensively (and loves the tidyverse) the tidyverse is so much easier to teach to people without any programming experience and who have very little faith in their tech or maths skills (which was me as well).

The tidyverse just 'made sense' to me when I started using R for the first time a few years ago, and now I love using R and programming. On the other hand, some of my ex-classmates learnt base R (because that's what we were taught) and found it hard, didn't learn anything properly, and now still think R or other programming languages are opaque and hard.

I'm not particularly fussed if Statistics Profs prefer data.table to dplyr or base R to tidyr, I know what is easier to teach, understand and use for me and a lot of other ecology/bio students and people.

<comment><USER24> I concur with your sentiments having cultivated data science teams from the ground up with diverse educational backgrounds.

Programming in base R is more akin to assembly language and has accreted a babel of inconsistencies that make it difficult to teach and learn. Learning base R isolates you into a Galapagos island of academics who are either ignorant of the needs of data workers or too elitist to engage with those not in their priesthood.

Learning Tidyverse is a considerably better transition for learning other languages, frameworks, and libraries.

Functional programming is closer to algebra than indexing into data structures with magic numbers. I've found more success teaching functional pipelines of data structures using the idioms in Tidyverse as a general framework for data work than base R. Abstraction has a cost but for learning it is the appropriate cost.

I sense that much of this `monopolistic` fear mongering is really about feeling out of date.

</comment></comment><comment><USER25> I teach and consult on R and data science. I had the privilege of learning R from one of R's core developers. My students often ask why I don't use the Tidyverse. The answer is because I don't need to - I can do everything the Tidyverse does and so much more in base R.

This article only briefly touches on what I think is the biggest issue with the Tidyverse. The Tidyverse is incredibly limiting. The "tidy" workflow hides many details of the language, which leads many users to think all data has to be in a tidy "data.frame" (or now tibble) and organized just so. The functions the Tidyverse authors have chosen to implement are seen as the limits of R's capabilities. Every data problem has to be forced into the Tidyverse box (or it is impossible). I cannot tell you how many Tidyverse scripts I have seen where 90% of the operations are munging the data into the correct format for the Tidyverse functions, when the original data can be handled with a single base R function (e.g. lapply). Most Tidyverse users accept this as just the way R is.

Most R users who only learn the Tidyverse never hit its limits, and for them the Tidyverse is perfectly OK. Those that do either resign themselves to the perceived limits of R, or (hopefully) start learning some base R. One friend who took the latter path once exclaimed to me "Logical subsetting is amazing!?!" \- this is a foundation piece of the language. That she went 2+ years in the "Tidyverse" without even knowing it was an option was eye opening to me.

To its credit, the Tidyverse is empowering - with very little programming knowledge a beginner can do a lot of data science. However, the majority of these people get stuck as "expert beginners". Some of these become fierce advocates of the Tidyverse without truly understanding base R. Meanwhile, none of the truly advanced R users I encounter use the Tidyverse.

ed.typo

<comment><USER13> "Meanwhile, none of the truly advanced R users I encounter use the Tidyverse."

This is most likely because truly advanced R users have been using R since far before the Tidyverse existed. A whole new generation of R users is being brought up with the Tidyverse, so im curious to see how the situation will be in 10 years time.

<comment><USER25> This is not entirely true - many are of the same "generation" as me. To be clear, I don't actually consider myself an advanced user. To me, many of these advanced users are pushing against the limits of the language in a way you don't see with run of the mill data science.

That said, I have heard my mentor say "I don't get the point of <insert tidyverse function> \- I did this 20 years ago."

<comment><USER26> > That said, I have heard my mentor say "I don't get the point of <insert > tidyverse function> \- I did this 20 years ago."

Tidyverse has a tendency to promote "new" things without any references to what came before. If you listen to their talks they speak as if they invented functional programming and the idea of pure and tiny functions working together.

And it trickles down to the users. A lot of people who learned tidyverse first for example, praise the `purrr` package, but have no idea that something like `Map()` is in base R.

</comment></comment></comment></comment><comment><USER27> I was waiting for the critique... but I never quite saw it.

Imho, the data problem Tidyverse is trying to solve is basically the ones we face in a database. So, select, join, inner join and so forth. Show me all the rows in this datatable where the 4th columm is larger than the 6th column and the number itself is odd. Something like that.

There might be other ways to do it, but you want your select, filter, summarize, mutate etc functions to all work with each other, pipe to each other and be compatible.

Maybe there is a better way to do all this -- I haven't seen it but I am not an expert -- but you have to show that to me.

So, in base R, walk through a set of example of mutating, joining, filtering and so forth, and show me how they are all easier. Then I'll say, wow there is an alternative to this Tidyverse thing. But in lieu of that demo, this felt more like an intro to a complaint than an actual complaint.

Edit: Also, its funny that Wickham is (apparently) such a nice fellow that people go out of the way to be nice to him in critiques.

<comment><USER28> His critique is more about the impact of the full ecosystem effect of the Tidyverse, not what you are referring to, which is just the dplyr semantics. The Tidyverse demands that it's many related packages use tidy data principles and lock users into that approach, which differs from base-R. Much of this discussion is really just a debate about dplyr and magrittr rather than the fragmentation that the broader tidyverse has brought on. All that said, I agree with many commenters that the Tidyverse's improvements to speed of development can more than offset the speed of execution issues, at least for small-to-medium datasets.

<comment><USER29> I think any one making the point of speed of development have seriously missed the boat. Lets be real: tibbles suck. Once you get the hang of data.table syntax for matrix operations, its superiority becomes impeccably clear.

I run a data science group a large geospatial company and we develop day in and day out in R and python. We've purged tidyverse as much as possible from all of our code base. We've moved completely over to data.tables, which make the vast majority of the tidyverse irrelevant.

<comment><USER30> Let's keep the discussion civil please. People have legitimately different needs, and just because a package isn't well suited to your needs doesn't mean that it doesn't help people with different backgrounds and goals.

</comment></comment></comment></comment><comment><USER31> Wow, the idea that the Tidyverse is somehow burdened with an overabundance of computer science or functional programming is just hilarious to this programmer who has had to maintain, debug and optimise other people's R code.

<comment><USER32> I think this is what the author is getting at, that correct tidyverse usage requires greater experience and knowledge and creates difficult to debug and optimize code when used incorrectly (and is easy to use incorrectly).

I tend to agree. I've found tidyverse code has a write-only quality to it. Since I'm going to see more of it I plan to dive into the inner workings of at least dyplr and purr.

That said it is hard to deny what Hadley Wickham has done for the R community and I can't write off the idea that he sees a bigger picture I'm missing.

<comment><USER24> Tidyverse code is not write-only; it is designed to be mostly read-only where the level of abstraction is at the level of the domain, which in this case is data frames and data pipelines akin to the same constructs in relational algebra/SQL or data processing (map/reduce).

Correct usage requires learning the right abstractions, but fortunately these abstractions are shared across language communities and frameworks.

If you are doing any data processing work and you do not know about foundational functional concepts ie map/reduce then I would argue the code you write is _less readable_ , overly focused on the idiosyncracies of how to do something rather than how entities and data are related to each other - <USER2> logic and structure.

The main optimization is to be correct and expressive of one's intention and purpose. If you need performance use Julia.

<comment><USER31> It's very hard to write expressive, readable code that munges some horrible tabular format into another arbitrary tabular format, to be fair. That said it's true that most R code doesn't seem like map/reduce, or some other pipeline of transformations. It's usually more like someone cut and pasted a long, hard REPL session into a notebook and has no intention of ever scrolling back up.

</comment></comment></comment></comment><comment><USER4> I initially viewed this article with skepticism, rather liking the functional style of the Tidyverse.

However, the claims the author makes are entirely valid. Data table is a fantastic package, and easy to learn. Tidyverse really has split the R community in disparate sets of R users.

To my mind, one of R's greatest strengths is its meta programming capabilities. It is this that allows such broad paradigms to exist within the same language. In that, the challenges that R is presented with is similar to languages like Lisp dialects and Scala.

I don't have solutions, but the author has convinced me that a problem is indeed there.

</comment><comment><USER33> I guess I am what you would call an R power user.

What's really crazy is I never realized how fast and easy excel pivot tables were to work with.

I know I know it sounds ridiculous.

But if you do a lot data splicing and dicing, excel can actually get your cuts out way faster through pivot tables than writing R code.

So if you use R for almost everything, give Excel a try as well. And the nice thing is this is even more powerful if you work with non data science folks, because then you can say I just did this in excel (where excel skills should be required by everyone in the company, and R for the data scientists) and they'll probably just leave you alone the next time they need something since they will try to do it in excel first.

<comment><USER27> Agree 100%, but only if it is a one time activity. If you have to automatically pull files and do the operation several times, it is better to go with R (or python, or awk sed or whatever).

</comment></comment><comment><USER34> Most of these tidyverse vs. data.table arguments end up sounding a bit irrelevant to me. They tend to focus on syntax, which is largely down to preference, or performance, which is really only important once in a while. And they seem oddly focused on the need to choose one or the other, when it seems obvious that all are good choices with various tradeoffs.

I tried my hand at saying something more original about this debate recently: [https://unconj.ca/blog/for-and-against-data- table.html](https://unconj.ca/blog/for-and-against-data-table.html)

On the other hand, it's nice to see some criticism of RStudio and Hadley, even if it does stray into the conspiratorial. They've done a lot for the R community, but they do have some obvious blind spots.

<comment><USER35> It's not that RStudio has nefarious motives that cause harm, it's that they have neutral and sometimes benevolent motives with unforeseen consequences.

</comment></comment><comment><USER36> At least in my case, although I do like some of the tidyverse (ggplot is tidyverse after all, and probably my favorite plotting system in any language), I find the "tibble" data structure and its disdain for row names really annoying. Many, many, packages which I use expect a data frame with row names and likewise output such. So I am constantly converting back and forth between tibbles and data frames.

<comment><USER18> The claim that ggplot2 is tidyverse can be disputed. It existed before and works fine outside of it.

<comment><USER30> You are of course free to dispute what is in the tidyverse, but I pretty strongly believe that it is part of the tidyverse

<comment><USER29> Seeing as the tidyverse is pretty much your invention, its well within your purvue to define what is in and what is out. I do think, respectfully, that both the op and the general sentiment article have the right of it.

Data.table and data.table-esque notation represent such an improvement of tibble/dplyr, that within my company, we're making a concerted effort to purge all tidyverse packages from general use (less ggplot). When new developers come on, if they are coming from tidyverse, their first task will be something involving pipes and data.table. Tidyverse was fine in school. It doesn't pass muster in production, at least not in our work.

Data.table syntax is simpler, easier to read, easier to teach, and orders of magnitude faster. It plays nicer with other packages than the tidyverse (if it fits into a DF, it almost always fits into a DT, and i've never met a tibble that I didn't wish was a data.table), and since almost all of our datasets are 10's to 1000's of millions of lines long, the decision was really made for us.

<comment><USER37> "Tidyverse was fine in school. It doesn't pass muster in production...."

This is a bit disrespectful.

"Data.table syntax is simpler, easier to read, easier to teach..."

This is rather arbitrary, and I don't think it's the majority view of the community, whatever the advantages of data.table.

"...orders of magnitude faster"

This is an exaggeration in most real cases, even according to the benchmarks pointed to by data.table.[1]

[1] [https://h2oai.github.io/db-benchmark/](https://h2oai.github.io/db- benchmark/)

</comment></comment></comment></comment></comment><comment><USER38> I am a feverent user of data.table and try to avoid dplyr when ever possible. I have colleagues who are exactly the opposite. No one cares.

If you want efficient code, try not to use dplyr. If you want readable code, data.table isn't the best answer. If you want speed, there are better languages than R to choose from, even with Rcpp.

If you want to get something done, go with whatever you know best and will do the job.

Avoid Stata.

</comment><comment><USER39> I recently tried to do some stuff in R with tidyverse, and was not a fan. I'm no expert, but the tidyverse's frequent use of nonstandard evaluation drove me crazy. It makes it so much more difficult to write functions encapsulating tidy functions. However, I never see people complain about this, so maybe I'm doing something wrong or not grokking something...

<comment><USER40> There is a new operator, {{ or "curly curly", aimed at dealing with this. It's part of the rlang (0.4) package and looks snazzy / easy to grasp.

Check it out: [https://www.tidyverse.org/articles/2019/06/rlang-0-4-0/](https://www.tidyverse.org/articles/2019/06/rlang-0-4-0/)

<comment><USER29> Yuck.

<comment><USER40> Have anything more constructive to say?

</comment></comment></comment></comment><comment><USER15> A friend pointed out to me that the author a few weeks ago put out a Python versus R comparison where he makes the rather incredible claim that the addition of some opinionated packages in the R ecosystem is more disunifying than the 2.7 to 3.x split in Python:

[https://github.com/matloff/R-vs.-Python-for-Data- Science](https://github.com/matloff/R-vs.-Python-for-Data-Science)

</comment><comment><USER41> When I'm writing the book (www.anotherbookondatascience.com) I always try to avoid the usage of tidyverse and got some criticism. But now I feel supported.

</comment><comment><USER42> at the end of the comments of the post [http://varianceexplained.org/r/teach- tidyverse/](http://varianceexplained.org/r/teach-tidyverse/)

compare base R:

      var1 <- "pounds"
      var2 <- "wt"
      mtcars[[var1]] <- mtcars[[var2]]/1000

Now hadley solution comment: You can now express that sort of thing fairly elegantly with dplyr:

      var1 <- sym("pounds")
      var2 <- sym("wt")
      mtcars <- mutate(mtcars,!!var1 := !!var/100)

     Watch your steps: !!, :=, sym.

Thats a lot of added complexity to be able to use column names, and to top it all, the use of non standard evaluation make things very complex when one go a little away from basic EDA.

<comment><USER8> Well, strictly literately to this case the tidyverse command would be:

mtcars %<>% mutate(pounds = wt / 1000)

Which is obviously simpler, and quite likely what a beginner is actually trying to do.

The real complaint - "tidyverse doesn't let us use variables to name columns!" \- is fair enough, following quasiquotation is hard work. If you need that, drop back to base R. However, it is harder to teach because all the common operations (select, filter, etc) will involve some combination of []/[[]] and relatively hard to figure out which one a piece of code is trying to do.

Except for the lucky few who have brains wired to think in terms of arrays and database relations, someone learning the language is unlikely to thank you for that.

</comment></comment><comment><USER43> Tidyverse was nearly the only thing I really liked about R. The syntax is quirky, it's full of cruft, and unpleasant to make full programs out of. But if I used a new package, I knew instantly how to use it.

</comment><comment><USER44> With respect, its laughable to think that anyone working on big-data with R has _not_ heard of `data.table`. When your livelihood depends on it, you tend to find it out one way or the other.

However, I'd claim most folks don't deal with big data or even medium data! [Aside: I chuckled at the 1e5 columns on the plot]. Personally, at the small- to-medium range, all languages are fast enough. It boils down to ergonomics for me. I _detest_ the subscript notation. It slows me down -- not just productivity-wise but also how I read or reason-about my code. Not surprisingly, I prefer `dplyr` over `data.table` (base R or `pandas`)

</comment><comment><USER45> One reason dplyr was promoted is that you can connect to different server backend like sparkr, sql etc, which is the enterprise direction RStudio aiming at. In the other hand, data.table is your friend when you are using your own machine. With more and more memory and cpu power available, its benefits are actually increasing.

I have been using data.table from almost day one, it does worth more recognition. The syntax is a little bit hard in the beginning but can be grasped after some efforts. I often feel that my exploration code will be much more tedious look if written in dplyr style.

<comment><USER46> If my university uses Enterprise RStutio on grid it is better to stick with tidyverse rather than data.table?

<comment><USER47> It doesn't make a difference. Rstudio's products aren't integrated with their libraries in particular.

</comment></comment></comment><comment><USER48> dplyr is an "advanced" topic?

It's, like, the easiest data manipulation library in any language.

<comment><USER15> People who were big into R pre-Tidyverse have this tendency to view the way they manipulated data frames pre-Tidyverse as the "basics," and Tidyverse as the "advanced" method. I think it does anyone learning R for the first time an incredible disservice, for the reason you state: dplyr is fantastic. But the old guard is (rightly) afraid that if people start with dplyr, they'll never get around to learning what base R provides for manipulating data frames. Which is good in my view, and nobody should have to learn that.

</comment></comment><comment><USER49> I think the same thing is happening in Python for data science due to Pandas. It's a great package, but many data scientists are only able to manipulate data using Pandas, and know little "base" Python.

<comment><USER50> That sounds incredible to me as using pandas without falling into the traps of chained indexing seems so much harder than 90% of Python.

<comment><USER51> I've been planning to go from "can work it out with copious examples" to "knows when to use apply, transform, etc. off the cuff" level of knowledge in Pandas - do you have any suggestions on good resources on this?

I'd like to understand better why the data structures work the way they do and thus have an intuition on what operations to use when. The O'Reilly Python Data Science Handbook[0] seems like it might be useful here, but I'm not sure if it is still up to date.

[0] [https://www.oreilly.com/library/view/python-data- science/978...](https://www.oreilly.com/library/view/python-data- science/9781491912126/)

<comment><USER52> I find the pandas documentation pretty good [0]. Actually, I find the documentation, official tutorials of pandas, numpy, matplotlib all pretty good. Each package has their own idiosyncrasies but I think the documentation and code examples cover them well enough.

The OReily book should not be that outdated if at all. It also covers other essential tools.

[0] [https://pandas.pydata.org/pandas- docs/stable/getting_started...](https://pandas.pydata.org/pandas- docs/stable/getting_started/index.html)

</comment></comment></comment></comment><comment><USER42> A related post is [http://varianceexplained.org/r/teach- tidyverse/](http://varianceexplained.org/r/teach-tidyverse/)

</comment><comment><USER42> What I see clearly is that is not a good option to begin with R inferno, except when you are a fighter prepared to learn at any cost.

</comment><comment><USER53> > Ironically, though consistency of interface is a goal, new versions of Tidy > libraries are often incompatible with older ones, a very serious problem in > the software engineering world.

I was (sadly) amused by a recent comment on HN linking to some tweets celebrating that some R code still ran four years later.

<comment><USER54> That was my comment, and the point wasn't that the R code still ran four years later--the point was that R Core's focus on backwards compatibility (in reference to R being hamstrung via its focus on "compatibility with S") isn't a bad thing. You're never going to get the 2.x - 3.x schism that python had with future version of R, assuming R core continues with that philosophy.

<comment><USER53> I completely agree with your point and I think saying that R is keeping compatibility with S doesn't make much sense. R is keeping compatibility with R, as it should.

Maybe I lack context, but those tweets looked to me as if someone said "I can run this four year old game in the latest release of Windows!" or "Three versions of Office later I can still open my excel files!"</comment></comment></comment>


CDC's canceled climate change conference is back on thanks to Al Gore - <AUTHOR> https://www.washingtonpost.com/news/energy-environment/wp/2017/01/26/cdcs-canceled-climate-change-conference-is-back-on-thanks-to-al-gore/ <comment><USER2> >Emails sent to participants and scheduled speakers did not explain the reason behind CDC's decision. Nor did the agency offer an explanation in response to a request for comment from The Washington Post, saying only that it was exploring the possibility of holding the event later in the year.

>The meeting now planned for Feb. 16 will take place outside of any government circles. Rather than at CDC, it will be held at the nonprofit Carter Center in Atlanta. It will be a one-day event rather than the three days originally planned.

It saddens me when government suppresses science.

<comment><USER3> I sorely hope that Trump will teach Americans to get over their misguided king-complex and realise that the president should be a figurehead with limited power, and an executive leader only during time sensitive (military) crises. The rest of the government should be governed by properly legislated remits, and answering to Congress, not the president. It's ridiculous and a recipe for disaster to leave so many policy areas to the whim of a single person.

<comment><USER4> Leaving it up to "the people" (read: Congress) over the past decade would've surely put us in a much worse position.

<comment><USER5> I voted for Obama, but I'm not so sure Congress would have been (much) worse.

<comment><USER4> I honestly don't understand this viewpoint.. the false equivalence lately is just insane.

We're talking about Climate Change, can anyone possibly believe that the Republicans in Congress for the past 6 years would have been better on Climate than what Obama achieved through executive orders? Do people just not pay attention?

Here's a quote from the official GOP platform[1]:

> _Climate change is far from this nation's most pressing national security > issue. This is the triumph of extremism over common sense, and Congress must > stop it._

Fuck it, I'll quote the rest of the references to climate in the official platform!

> _Information concerning a changing climate, especially projections into the > long-range future, must be based on dispassionate analysis of hard data. We > will enforce that standard throughout the executive branch, among civil > servants and presidential appointees alike. The United Nations' > Intergovernmental Panel on Climate Change is a political mechanism, not an > unbiased scientific institution. Its unreliability is reflected in its > intolerance toward scientists and others who dissent from its orthodoxy. We > will evaluate its recommendations accordingly. We reject the agendas of both > the Kyoto Protocol and the Paris Agreement, which represent only the > personal commitments of their signatories; no such agreement can be binding > upon the United States until it is submitted to and ratified by the Senate._

> _We demand an immediate halt to U.S. funding for the U.N.'s Framework > Convention on Climate Change (UNFCCC) in accordance with the 1994 Foreign > Relations Authorization Act. That law prohibits Washington from giving any > money to "any affiliated organization of the United Nations" which grants > Palestinians membership as a state. There is no ambiguity in that language. > It would be illegal for the President to follow through on his intention to > provide millions in funding for the UNFCCC and hundreds of millions for its > Green Climate Fund._

> _We firmly believe environmental problems are best solved by giving > incentives for human ingenuity and the development of new technologies, not > through top-down, command-and-control regulations that stifle economic > growth and cost thousands of jobs._

These people are pathological.

[1] - [https://www.gop.com/platform/americas-natural- resources/](https://www.gop.com/platform/americas-natural-resources/)

<comment><USER3> On net, long term, Obama will turn out not to have been very effective at all, because everything he's done through executive order has/will be undone through executive orders in the few weeks. That's the curse of executive order, and why good, long term legislation won't arise from executive order.

<comment><USER4> That's all well and good, but there is absolutely no indicaition that congress would have produced anything resembling "good, long-term" legislation.

In my completely made up system, if we started at 0 and add up the number of points in favor of averting Climate Change, Obama probably added 3 for a few years with the Paris Agreement, CAFE and some of the EPA rules were also pretty beneficial, maybe net 4 over his term. Call it +10 for Obama. Trump's going to reverse many of those decisions, probably make some of them much worse within the next few years. Let's say he wipes out all of Obama's work and does 50% more 'damage' in this system. So by 2020, we're at -5.

If we had left it to the Republicans in Congress, we would have been at least at -5 every year since 2010..

So -5 net with Obama/Trump's dueling executive orders or -50 with Republicans in Congress.

Yes the system is made up and the points don't matter, but there is absolutely no way we would be in a better position. It's the height of naivety to dismiss what they promised to do and to assume they would have suddenly changed their views 180º to support action on climate change for 'the limelight'.

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> It's good to see that this issue has expanded beyond the realm of politics. There is hope when non government organizations are able to put something together in the absence of federal participation.

</comment><comment><USER7> Thanks to the Carter Center here in Atlanta for hosting it as well! And all the foundations that pitched in.

</comment><comment><USER8> Al Gore 2020?

<comment><USER9> Haha good idea since the last 1990s Democrat retread did so well.

<comment><USER10> All I want for Christmas is a Democratic candidate whose name I didn't learn in grade school.</comment></comment></comment>


Why do so many Egyptian statues have broken noses? - <AUTHOR> https://edition.cnn.com/style/article/egyptian-statues-broken-noses-artsy/index.html <comment><USER2> Looks like the same article as here: [https://news.ycombinator.com/item?id=19388979](https://news.ycombinator.com/item?id=19388979)

</comment><comment><USER3> A: "The damaged part of the body is no longer able to do its job," Bleiberg explained. Without a nose, the statue-spirit ceases to breathe, so that the vandal is effectively "killing" it.</comment>


You Didn't Make It to Y Combinator or TechStars, Now What? - <AUTHOR> http://blog.fairsoftware.net/2009/04/07/you-didnt-make-it-to-y-combinator-or-techstars-now-what/

<comment> old-gregg We took our rejection 2 years ago as a wakeup call: If you can't sell yourself to YC, number one thing you need to learn is how to sell, period. Because "how to sell" part will start matter more and more down the road, approaching "how to code" in importance.

If you couldn't sell yourself to YC, you may not sell to your customers either, or other investors, or future employees, etc etc etc

Learn how to sell. Your idea is probably fine - 95% of YC-accepted companies are building tiny incremental improvements to existing products.

<comment><USER2> If you can't sell, get closer and closer to having a product or getting to having a business. Talk is cheap, but having made something matters more.

A basketball team will be more likely to draft you because you've de-risked yourself, you are probably already on your way to winning the NCAA.

I wrote a little about this before on my blog about getting into YC, etc.: [http://andrewlee.com/2009/03/01/startup-march- madness-2009-g...](http://andrewlee.com/2009/03/01/startup-march- madness-2009-getting-into-launchbox-and-ycombinator-techstars-etc/)

</comment></comment><comment><USER3> Your post really sounds like you massively misunderstand the program.

You mention using the $15K in funding to buy a graphic design,or to pay for some PR. The idea behind the $15K in funding is (as I understand it, never having directly participated) is that it is just enough to cover _reasonable_ living expenses. It is enough to take your mind off of "how will I pay my rent", but not enough to put your mind on "what clubs will I party at this weekend after I spend a day surfing (ocean surfing, not web surfing).

You make reference to Googleing obsessively on certain topics to learn. But how do you know that which you do not know? I've been involved in startups for years, and I still learned things in the very first Startup School ('05, I think), that I had never considered before. No amount of Google-fu would have been as efficient or effective. It is safe to assume that having 1-1 time with pg and his ilk would be massively more valuable than having direct access to Googles entire cache of startup-related data.

You also overly discount the value of what you label as "notoriety". No matter what you build, you have to get your message out there somehow. Being prefiltered as a valuable or insightful company can accelerate the growth of your product/company.

At the end of the day, YC and programs like it are not the only way to launch a company. But I think the value, especially to "entry level entrepreneurs" is pretty significant.

</comment><comment><USER4> _whenever you have the choice between worrying about what will happen, applying to a program, preparing a presentation, or making progress on your product… always choose the latter. Keep coding. Ignore the distractions. You'll end up ahead._

You've got to eat somehow. Not everyone's lucky enough to have the resources to code in isolation for months without thinking about business development. Plus when you finish coding and have to present, you've missed out on valuable practice opportunities ;)

<comment><AUTHOR> Does the $15K help you eat for any significant amount of time?

Let's say t will take a year for your startup to take off or make it to a funding event. For 3 people, you'd be splitting $415/month. Some people can make that kind of income with one day of IT consulting, so there _are_ workarounds to getting into YC.

Again, don't get me wrong: if you are accepted, by all means take the offer!

<comment><USER4> I didn't really just mean YC's money, more the unfeasability of ignoring all concerns but the code. Obviously some people (the IT consultant you mention, or those laid off with a nice severance, etc) don't really have their empty fridges as so much of a motivation ;)

I wouldn't really expect a startup to operate on $15k for a year; the whole point is you use it as a springboard for further funding, surely?

</comment></comment></comment><comment><USER5> Try AlphaLab <http://news.ycombinator.com/item?id=550892>

</comment><comment><USER6> interesting way to rationalize not getting in: "it isn't very good"

<comment><AUTHOR> Don't get me wrong. YC is good. But not getting in is not the end of the world. It's the difference between taking a shortcut or having to do the work yourself.

<comment><USER7> From quotes like this

"It's great if you are the kind who learns by sitting passively on a chair listening to someone."

it's clear you don't understand what happens within YC.

We don't have classes where people sit and listen to us talk about stuff that they could, if they were sufficiently diligent, find by searching in Google. Nearly all interactions we have with startups are about their specific situation: things like what features to build first, when to launch, what to say on the frontpage, how to deal with specific investors they're talking to, etc.

<comment><AUTHOR> Thanks for the clarification. I updated the blog accordingly with your comment.

I wasn't referring to YC specifically when I wrote the original paragraph, I had other incubators in mind. Sorry if it sounded too harsh.</comment></comment></comment></comment>


CoreOS Linux Alpha Remote SSH Issue Post-Mortem - <AUTHOR> https://coreos.com/blog/security-brief-coreos-linux-alpha-remote-ssh-issue.html <comment><USER2> Nice writeup. I think the main takeaway from this story is that always remember to write tests for error handling in addition to "happy cases".

</comment><comment><USER3> Thanks, but...Alpha software. Am I wrong in thinking of this as an overreaction?

<comment><USER4> There's an explanation of why we care about alpha releases in the article - does that cover it for you?

<comment><USER3> Not really -- but I think that's because it's not clear/I don't understand (latter is more likely) what role the alpha channel plays in CoreOS deployments.

I _think_ I understand that you want us to use deployments of Alpha software as a smoke test in production environments. I'd be interested in hearing what risk profile you're expecting users to fit when deploying the Alpha channel in production systems, because it's completely alien to my way of thinking about stability in large scale systems. Today it's an SSH vulnerability, tomorrow it could be something worse. Who is going to take these risks in production?

<comment><USER4> There's no strong reason to believe that the alpha releases have any worse security than other releases in the general case - the vast majority of security issues we've had to deal with are discovered months or years after they vulnerable versions first shipped, and so apply equally to stable.

I don't think users should have entire deployments of the alpha releases, but I _do_ think that one of the strengths of distributed computing is that you can run a subset of your deployment on alpha without worrying that bugs are going to take down your entire deployment. That makes it much easier for you to have confidence that new stable releases won't break things for you, which means you can adopt stable releases more quickly and avoid the security risks inherent in running older versions of software.

For that to be possible you have to be confident that we're not shipping alpha releases with gaping security issues, and so it _is_ a big deal when we fail in that respect.</comment></comment></comment></comment>


Strongest chess player, ever - <AUTHOR> http://en.lichess.org/blog/U4mtoEQAAEEAgZRL/strongest-chess-player-ever <comment><USER2> This is extremely impressive - it's cool that talented programmers are pushing the limits of computer science to advance the state of the art of chess engines.

However, I also wish that there were comparable efforts to create AIs that train humans. Basically, figure out a way to systematically, efficiently, and scalably train amateurs into masters. That IMO would be absolutely amazing (and something I'd gladly pay for).

<comment><USER3> In my opinion, this would be a lot more difficult. I am slightly naive as to exactly how stockfish works, but:

Computers can do a lot of accurate brute-forcing; humans must see the position in a more holistic, intuitive way.

Excellent human players and excellent computer players are presumably doing completely different calculation tasks.

I would suggest that computers are _still_ bad at approaching the task in a human-like way, but they will always be able to improve their method via Moore's law (at a minimum) where humans are stuck at their current level.

Stockfish might be able to tell you what it was doing, but not in a way that it would be reasonable for a human to follow.

What you are looking for is a teacher. You can pay for those ;) The closest we have got to a teacher app is perhaps a MOOC: not much computational progress has been made.

<comment><USER4> > Stockfish might be able to tell you what it was doing, but not in a way that > it would be reasonable for a human to follow.

I (a non-chess player) just tried to play a game against the highest level AI (and lost, obviously).

Doing an analysis of the game afterwards, this is exactly what I experience: I do "f4" and I'm told (through the analysis tool) that the best move was "Nf3".

Now, the obvious question this leads to is: why? Why was this a better move? I don't think that, as a human being, memorizing "best moves" is going to lead to much improvement: we need to know WHY that move was the best move.

I'm sure there is a human-friendly way to explain why one move is the best move, and why my move wasn't, but the computer probably doesn't know this explanation, because it's approaching it from a brute-force perspective.

Surely, a chess computer can brute force all possible combinations, and deduce that this was the best move. But when this is not possible for a human being, just informing the me that "what you just did was not the best move", doesn't really do much to help me (as an amateur player).

The game, for reference: [http://en.lichess.org/ehWjHnIc#0](http://en.lichess.org/ehWjHnIc#0)

<comment><USER5> "f4" creates a hole on "e4" in your pawn structure. This hole becomes a valuable outpost where your opponent can position a knight (or other piece) without it being harassed by one of your pawns. This means to get rid of it, you need to trade a piece for it, and when your opponent recaptures, it will give them a passed pawn.

"Nf3" protects your d4 pawn and also threatens the e5 square. Probably most importantly, it clears a piece so you can castle kingside.

</comment></comment></comment></comment><comment><USER6> What I would love to see is work on chess engines having better strength levels for amateurs. For me at least there is a line where everything below a certain level I can beat 95 percent of the time and everything at or above that level I lose to 95 percent of the time.

<comment><USER7> One of my long-term goals is to write a chess program that plays more like a lower-level human. Every program I've played seems to have the same pattern: play like a master for ten moves, then make an insultingly obvious blunder to let me catch up. Lather, rinse, repeat. It's neither realistic nor fun. Might as well play with a handicap.

Real players at each level tend to have _characteristic_ weaknesses that are expressible in terms of the same factors used in a program's evaluation function. Consider some of the following, which players at a certain level will exhibit and then get past as they improve.

* Bringing the queen out too early.

* Missing pins and discoveries.

* Failing to contest the center.

* Creating bad bishops.

* Bad pawn structure.

* Failing to use the king effectively in the endgame.

These are all quantifiable. They could all be used to create a more realistic and satisfying opponent at 1200, 1500, 1800, etc. All it takes is some basic machine learning applied to a corpus of lower-level games, and a way to plug the discovered patterns into the playing engine.

</comment></comment><comment><USER8> note that it's just a chess "engine" and doesn't come with a human-usable front-end.

But it's not hard to get the thing to use xboard in linux.

This is what i use to invoke it: xboard -fUCI -fcp stockfish -sUCI -scp stockfish&

it's very challenging - and in my experience, will unfortunately peg a core unless you explicitly pause the front end (the "P" button in between the two arrows in the upper right)

<comment><USER9> what do you mean? just click play a game

[http://en.lichess.org/](http://en.lichess.org/)

<comment><USER8> Yes that way also works!

But the xboard way allows me to modify the engine, get debug output, try various different weights for pieces, put two engines against each other to see how specific features affect game play...

It helps me demystify such sophisticated software.

<comment><USER9> heh, did you ever upload a match between two engines to the site and let the online engine analyze it?

<comment><USER10> Very interesting idea!

</comment></comment></comment></comment></comment><comment><USER11> _> During the final event, after playing 64 games against Komodo, Stockfish won with the score of 35½-28½. No doubt is further allowed: Stockfish is the best chess player ever!_

From a statistical point of view, this isn't actually significant, despite the fact that draws help reduce the variance.

45 of those games are draws, leaving a 13-6 score in favor of Stockfish. Considering a null hypothesis of a binomial distribution with n=19 and equal chance of winning, the two-sided p-value for that score is 0.115. Unless you already have strong evidence that Stockfish is better than Komodo, you shouldn't conclude anything about which one is best.

<comment><USER12> But it fundamentally _isn't_ binomial across 19 games, because of draws. You can't just ignore draws from the analysis, to do so is terrible application of statistics.

<comment><USER11> Once you condition on the number of draws, you _do_ get that binomial distribution.

Suppose you have a coin, which gives a random outcome X. But you can only observe the outcome of X when another independent binary random variable Y is true. How can you tell if X is biased? Since X and Y are independent, the observations where Y is false are irrelevant since they don't tell you anything about X. So you just keep the observations where Y is true, and from there you can apply a binomial statistical test to the observations of X.

[ In case you're wondering whether applying statistical tests to variable sample sizes is valid, the answer is yes: a p-value is a uniform random variable from the set of observables (augmented by a continuous random variable, since our set of observables is discrete) to [0,1]. Our p-value is a mixture of p-values on smaller sample sizes, so it is still uniform. ]

This is exactly what happens here: consider a random outcome {win,lose,draw}. If you don't have a draw, let Y be true and X be the outcome of the game. If you have a draw, let Y be false and X be a random coin with the same distribution as for non-drawn games. Then X and Y are independent random variables and the above applies.

Informally: draws are not useful information in determining whether there are more wins than losses.

<comment><USER13> I'm not sure that discounting draws is the right thing to do either. For example, Petrosian was not the strongest attacking player but was very, very tough to beat.

This also calls into the question the notion of "strongest chess player." Who is strongest, the flashy attacking player that wins half the time and loses the other half, or the stonewall that poses little threat but that you can never beat?

</comment></comment></comment></comment><comment><USER4> Has anyone else watched [http://en.lichess.org/tv](http://en.lichess.org/tv)?

If this is two humans playing against each other in real-time, that is really impressive! It's so mind-boggling fast (mind you I'm not a chess player).

<comment><USER14> ChessNetwork on YouTube[1] has many videos of blitz chess tournaments where he provides commentary as he plays. Very interesting to watch!

[1] [https://www.youtube.com/watch?v=7YWYS209ydE](https://www.youtube.com/watch?v=7YWYS209ydE)

<comment><USER15> Wow, very cool!

Can someone explain what's going on at 20:24 : [http://youtu.be/7YWYS209ydE?t=20m20s](http://youtu.be/7YWYS209ydE?t=20m20s) ?

He forks the king and the queen, then his opponent moved his queen!?! This is not a legal move right?

<comment><USER14> Had to re-watch a couple times: on that move, white had already moved pawn from g2 to g3, the knight from d6 to e8 is a pre-move that happens very quickly (you can see the red square); while white was setting up that pre- move, black moved the queen.

Does that clarify or did I miss what you are asking?

<comment><USER15> Got it, thanks!

</comment></comment></comment></comment></comment><comment><USER16> Can anyone comment on what makes stockfish different from other chess engines? If I'm curious about state of the art in computer chess, is it worthwhile to study its source? What interesting ideas should I expect to see there beyond what I vaguely know to be the standard approach from introductory AI courses, i.e. some sort of alpha-beta pruning search?

<comment><USER17> You should indeed study Stockfish's source, not because it is the strongest computer chess engine, but because it is exceedingly well-written. Most engines that are public are a big hacky mess, but the Stockfish maintainers have made cleanliness a really high priority (sometimes proposed changes that increase the engine's strength are rejected because they're too much code), and it has clearly paid off.

<comment><USER18> >sometimes proposed changes that increase the engine's strength are rejected because they're too much code

that seems insane.

<comment><USER17> I know, doesn't it? But sometimes someone finds a way to simplify the patch so it's less code and almost as effective, and sometimes future improvements are made easier because the existing code is easier to work with. And it's hard to argue with the results.

</comment></comment></comment></comment><comment><USER19> I bet it would be unnerving for someone like Kasparov or Magnus Carlsen to play this program, where it would have 1 minute on the clock and they could have the whole day. It would make many of the moves in under a second and they'd be better than the grandmasters' moves!

<comment><USER20> You can watch Magnus Carlsen play a chess program on his phone here:

[https://www.youtube.com/watch?v=pNvVWeHZG00](https://www.youtube.com/watch?v=pNvVWeHZG00)

As you can tell from his commentary, he thinks the machine is rather stupid (and he is winning after the opening) but the machine is a much better calculator than he is and when the situation becomes more concrete he has to force a draw.

Of course, the computer on his phone is considerably worse than the best chess engines, but top chess players generally consider computers to be excellent calculators but dumb in terms of general strategy.

<comment><USER21> > but top chess players generally consider computers to be excellent > calculators but dumb in terms of general strategy.

Is computer assisted chess a thing? Perhaps with standardized hardware, but any software.

In chess I'm good at strategy but terrible at calculating and I miss obvious stuff all the time in my fight for strategy. I always thought I'd do great with computer assistance to look for the obvious stuff, and me telling the computer the long term strategy.

<comment><USER22> > In chess I'm good at strategy but terrible at calculating

Chess is 99% tactics/calculation. What we call "strategy" is just a set of heuristics that we use to avoid having to do endless calculations. However, a lot of those heuristics are already included in most chess playing software. So, if you're weak player as a whole, even if you have some strategy acumen, your contribution in an assisted chess setting will be negligible. The computer will be doing all the work anyway.

My rating is around 2000 and I have done some assisted chess playing and I can tell you that it's extremely hard to not just take computer's suggestion at every move. The chance that I'll come up with some brilliant move that the computer missed is very slim.

<comment><USER21> I think I misunderstood what "calculating" means in a chess setting. I thought it meant checking the current position of the pieces and making sure you are not about to be attacked.

But googling it suggests it's more about thinking of the value of each move relative to others. If that's the case I'm not actually bad at that.

> I can tell you that it's extremely hard to not just take computer's > suggestion at every move.

Is that how it works? The computer just basically plays and shows you some moves it likes?

That's not what I meant, I was thinking that you tell the computer something like: I want to capture piece X using Y 10 to 20 moves from now, perhaps by going via this direction. Tell me the best series of moves to get there while avoiding traps.

Or even better give it 2 or 3 such scenarios and have it tell you how dangerous each one would be so you can pick one.

Basically really narrow down the permutations the computer has to calculate.

</comment></comment></comment></comment></comment><comment><USER23> Stockfish 5 was released yesterday: [http://stockfishchess.org/](http://stockfishchess.org/)

As mentioned elsewhere in this thread, Stockfish is just an engine - you must install a GUI separately. XBoard is well known, but there are better alternatives:

[http://chessx.sourceforge.net/](http://chessx.sourceforge.net/)

[http://scidvspc.sourceforge.net/](http://scidvspc.sourceforge.net/)

</comment><comment><USER24> I always thought of the heuristics for evaluating a chess position as the really hard part of building a chess engine; i.e. how do you capture all of the positional subtleties in a number to feed into minimax? But looking at the source, it's not really that complicated [1]. Can someone who knows more than me comment on that? Is it that the innovations are elsewhere? That good chess really can be boiled down to < 1000 LOC? That the numbers in this heuristic are just super expertly tuned?

[1] [https://github.com/mcostalba/Stockfish/blob/master/src/evalu...](https://github.com/mcostalba/Stockfish/blob/master/src/evaluate.cpp)

<comment><USER25> Nearly all the evaluation terms in Stockfish have been extensively tuned. Joona Kiiski used the SPSA[1] algorithm on a lot of them. Others have been hand-tuned using tens of thousands of games per attempt on fishtest [2]. Fishtest actually just recently got support for running SPSA tuning as well.

There is also a strong bias towards simplification, so if an evaluation feature is not proven to be an improvement it will be removed. Over the last few Stockfish versions, the # of lines has actually decreased in each version.

[1] [http://en.wikipedia.org/wiki/Simultaneous_perturbation_stoch...](http://en.wikipedia.org/wiki/Simultaneous_perturbation_stochastic_approximation) [2] [http://tests.stockfishchess.org/tests](http://tests.stockfishchess.org/tests)

<comment><USER24> That's helpful, thanks.

</comment></comment></comment><comment><USER26> Are chess engines still trying to play against humans in an interesting way? (I understand they beat human players, but that people feel computers play in dull ways).

Is there a Turing Test for computer chess, where humans and computers play each other and they, and commentators, analyse the play, but no-one knows who is a computer or human until after the commentary is published?

And if we ignore humans are people playing computers against other computers for some kind of machine learning play?

And how optimized for speed is the software? Do they really crunch out all performance they can?

(Sorry for the barrage of questions but I don't know enough about this space to do efficient websearches).

<comment><USER27> > I understand they beat human players, but that people feel computers play in > dull ways.

Without knowing anything about chess, I'd say that makes humans sounds like sore losers. I wonder if this will lead to having tournaments where you don't get points for winning matches but rather get judged on style by a panel of (human) judges...

<comment><USER28> In the 1800s, most chess was much more aggressive, and it was frowned upon to reject an offered exchange of pieces. Consequently, games were faster, moves were riskier, and everything was arguably more interesting.

In other words, modern humans have been playing in dull, more reliable ways for over a century, so even if not sore loserish, it's at least a bit like the pot calling the kettle black.

</comment></comment></comment><comment><USER29> TCEC is cool but has less statistical power than many ratings lists out there, which show that Houdini, Komodo and Stockfish are very closely matched, with Houdini having a slight edge at long time controls and a moderate edge at quick time controls. Stockfish does release more frequently and I'm not sure which version competed in TCEC, but until the lists catch up this article is fluff.

<comment><USER30> Houdini had a slight edge against Stockfish DD, which is a few months old. The latest Stockfish version (v5, very similar to the version that competed in the TCEC Superfinal) is ~60 Elo stronger, and demolishes Houdini at all time controls. It should soon start hitting the more conservative ratinglists.

<comment><USER29> "Demolishes"? Hardly. It is so far tied with H4 on IPON. Even if it winds up on top, it will not be by a large margin.

</comment></comment></comment><comment><USER31> (Side note: what the hell is that website trying to do to my computer? It's trying to open so many connections on weird ports!)

<comment><USER32> Looks like they're using a bunch of websockets for something. Probably that live player count up in the navigation.

</comment></comment><comment><USER33> I'm rather surprised at how relatively simple and small the codebase is: [https://github.com/mcostalba/Stockfish/tree/master/src](https://github.com/mcostalba/Stockfish/tree/master/src)

<comment><USER34> Especially interesting to see everything hardcoded into C++. Quite different from most modern codebases that use a mix of languages, configuration files, etc.

</comment></comment><comment><USER35> Just try havin fun.. [http://en.lichess.org/SMVJP07p](http://en.lichess.org/SMVJP07p) The strongest player.. me kinda noobs..

</comment><comment><USER36> lichess doesn't seem to be giving it enough juice to perform at its stated levels at the moment.

On my first try I managed to draw the highest AI level, rated at 2510, while my rating is under 2000 irl. (I was unable to find an "offer draw" button so relied on the 50-move rule) Against stockfish running on my PC that would be impossible. [http://en.lichess.org/reDfuSvI](http://en.lichess.org/reDfuSvI)

<comment><USER37> Have you actually read the article? It's all explained [http://en.lichess.org/blog/U4mtoEQAAEEAgZRL/strongest- chess-...](http://en.lichess.org/blog/U4mtoEQAAEEAgZRL/strongest-chess-player- ever)

<comment><USER36> I did. I was complaining about the rated strength of the AI levels, they're off by a wide margin.

</comment></comment></comment><comment><USER38> That site's pretty impressive. They're open source and give away for free what Chess.com charges for.

</comment><comment><USER39> I don't think that the present situation, when the top chess playing program is free and open-source, is good for innovation.

I estimate that it would take me six months of work to get to the top-20 in the world, and I don't see how I can justify that work to myself.

<comment><USER26> I did not downvote you.

Your comment seems either ignorant of the amount of work involved, or very arrogant about your skill, but does not provide any information for me to judge.

Why do you think you could beat even rybka with just six months of work?

<comment><USER39> Rybka is better than top-20, so I did not claim that.

I was a half-pro ~10 years ago. I won the championship of my country. I wrote an M.Sc. thesis on evaluation tuning. If I'd use the standard approach that I already know, 6 months is a conservative estimate. The question is - what for?

<comment><USER40> >The question is - what for?

Uh, to win prize money?</comment></comment></comment></comment>


Ask YC:  How's the tech job market treating you? - <AUTHOR>

I've heard a lot of talk about unemployment recently in the news, but I don't know too many people that are struggling to find work here in the Valley.  Maybe I'm in a weird pocket, but even the consultants around me seem pretty busy.<p>I'm interested in hearing tales from the trenches.  How's the job market treating you? <comment><USER2> I have several friends who have recently been laid off from their software development jobs (most, but not all in silicon valley); but with the possible exception of one person I'm not sure about, they all found new positions within a few weeks.

I suspect that -- like in other economic downturns -- most of the people who lose their jobs and stay unemployed for an extended period of time will be people who never should have had a job in the first place.</comment>


Evidence of 90M-year-old rainforest uncovered beneath Antarctic ice - <AUTHOR> https://www.nbcnews.com/science/environment/evidence-90-million-year-old-rainforest-uncovered-beneath-antarctic-ice-n1175021 <comment><USER2> I wonder what fossil treasures are sitting beneath the ice in Antarctica. Most people don't realize how massive the continent is since it doesn't show up fully on maps - it is nearly 1.5 times the size of America.

If it was a wetter, warmer place millions of years ago, it would have been teeming with life.

<comment><USER3> Just to clarify because it was unclear to me and areas are unintuitive with map projections, it's 1.5 times the size of the United States. The U.S is ~10 million km^2, Antarctica 14 million km^2, and the American continent 42 million km^2

<comment><USER4> I've always thought a few school subjects should be put off until kids are mature enough to "get" them.

One is history class (much more meaningful when you are old enough to understand human nature) and another is mercator projections.

I always thought Canada was so much bigger than the US but they're about the same size - 3.5m sq miles.

<comment><USER2> > I always thought Canada was so much bigger than the US but they're about the > same size - 3.5m sq miles

That's because the largest state, Alaska, appears like its a part of the Canadian landmass

<comment><USER4> I think this shows it best:

[https://upload.wikimedia.org/wikipedia/commons/e/ee/Worlds_a...](https://upload.wikimedia.org/wikipedia/commons/e/ee/Worlds_animate.gif)

</comment></comment></comment></comment></comment><comment><USER5> One explanation would be Earth's spin axis was in a different spot millions of years ago due to polar shift.

Related research:

[https://www.discovermagazine.com/environment/millions-of- yea...](https://www.discovermagazine.com/environment/millions-of-years-ago- the-poles-moved-and-it-could-have-triggered-an-ice-age)

<comment><USER6> True polar wander sounds like one of the most terrifying natural disasters I could imagine.

<comment><USER5> Gradual polar shift happens over millions of years; specifically characteristic rate of true polar wander is 1° or less per million years.

If the shifting occurs as a "Cataclysmic" polar shift [1] or geomagnetic reversal [2], the periodic reversal of the Earth's magnetic field (effectively switching the north and south magnetic poles) — I agree that it would likely have terrifying results for the inhabitants of Earth.

[1] [https://en.m.wikipedia.org/wiki/Cataclysmic_pole_shift_hypot...](https://en.m.wikipedia.org/wiki/Cataclysmic_pole_shift_hypothesis)

[2] [https://en.m.wikipedia.org/wiki/Geomagnetic_reversal](https://en.m.wikipedia.org/wiki/Geomagnetic_reversal)

</comment></comment></comment><comment><USER7> I was wondering how they knew it was climate change and not continental drift and it seems like Antarctica hasn't moved much in the last 100 million years.

</comment><comment><USER8> Now if we would only get to work on global warming we could restore the Antarctic jungle to its proper Mesozoic glory.

</comment><comment><USER9> Original paper: [https://www.nature.com/articles/s41586-020-2148-5](https://www.nature.com/articles/s41586-020-2148-5)

<comment><USER10> >"A climate model simulation shows that the reconstructed temperate climate at this high latitude requires a combination of both atmospheric carbon dioxide concentrations of 1,120–1,680 parts per million"

Does this allow for other methods of warming? Does it assume that all things would be equal except for carbon dioxide levels? If so, why?

</comment></comment><comment><USER11> On this topic, check out the recent PBS Nova: Polar Extremes which discusses the climatic history of the earth and findings such as this one.

<comment><USER12> You can watch it here: [https://www.pbs.org/wgbh/nova/video/polar- extremes/](https://www.pbs.org/wgbh/nova/video/polar-extremes/)

If you get blocked because of region; New York works.

</comment></comment><comment><USER13> Fortunately some of those genus survived up to our time, including the only parasite conifer known that is alive. Plants in those forests would look like this:

[https://www.junglegardens.org/blog-posts/2018/12/3/the- asian...](https://www.junglegardens.org/blog-posts/2018/12/3/the-asian- bayberry-nageia-nagi)

</comment><comment><USER14> Anyone else wonder what kind of horrors are currently frozen down there. I'm thinking smallpox and other problems we believed were gone for good.

<comment><USER15> Would an ancient virus be any harm to modern humans? Or would we be too different to what was around millions of years ago for it to be a le to survive and infect people?

<comment><USER13> I think so. We can be modern, but we still breath exactly like any other mammal, including prehistoric species. Our methabolic routes are shared with lots of other species, extant and extinct

</comment></comment></comment><comment><USER16> If Earth's ice caps' ice cover were to melt, how much landmass would that uncover? ice-cap - ice + water = ?

<comment><USER17> Antarctica is about 150% the size of Europe.

<comment><USER18> The combination of melted ice and thermal expansion would mean that a lot of it would be flooded.</comment></comment></comment>