PERF: don't materialize arrays on checking in groupby
```
    before     after       ratio
  [a6fcec6c] [6edcdc0d]
-   15.41ms     9.48ms      0.62  groupby.groupby_size.time_groupby_size

```

this is more dramatic as the size increases as this should be pretty much a constant time operation (the actual size computation itself).