groupby pd.Grouper with freq gives inconsistent groupings
#### Code Sample, a copy-pastable example if possible

```python
dates = pd.date_range(start='2020-02-01 09:30:00.01', periods=5, freq='14ms')
dates2 = pd.date_range(start='2020-02-01 11:00:00.01', periods=5, freq='8ms')
df = pd.DataFrame({'A': [1] * 10 + [2] * 10,
                   'B': np.concatenate((dates, dates2, dates, dates2)),
                   'C': np.arange(20) + 1})

df["B"] = pd.to_datetime(df["B"])
df['num_group'] = df.groupby(["A", pd.Grouper(key="B", freq="60ms")]).ngroup()
df
```
For the desired group of len=5, where A==1 and B about 9:30:
num_group==0 has length 4, num_group==1 has length 1

```python
df['num_group'] = df.groupby(["A", pd.Grouper(key="B", freq="61ms")]).ngroup()
```
num_group==0 has length 1, num_group==1 has length 4

```python
df['num_group'] = df.groupby(["A", pd.Grouper(key="B", freq="70ms")]).ngroup()
```
num_group==0 has length 3, num_group==1 has length 2

```python
df['num_group'] = df.groupby(["A", pd.Grouper(key="B", freq="72ms")]).ngroup()
```
num_group==0 has length 5 (expected output, this is all A==1 and B around 9:30)

```python
df['num_group'] = df.groupby(["A", pd.Grouper(key="B", freq="73ms")]).ngroup()
```
same split of the group as freq='60ms'

For freq greater than "100ms" (have not rigorously tested), appears to be the same (expected) as the 72ms freq case.

#### Problem description
Performing a groupby on a pd.Grouper with varying freqs returns differing ngroups() for some reason. As I change freq from 60 to 73ms, it splits what should be one group into subgroups of varying sizes.

Grouping on the pd.Grouper and then calling ngroup would be an nice way to perform secondary groupby-aggregations using the num_group as a key instead of a time window. If we have a sparse timeseries, then resample or rolling would 1) create additional unwanted rows or 2) condense the "B" time series column such that there isn't a unique key to merge back to the original dataframe.

Since the first five rows are from 09:30:00.010 to 09:30:00.066, I would not expect to have to use a freq >= 100ms to group these 56ms rows together, and the freqs cause unexpected groupings without explanation.

#### Expected Output
Expected output for all of the above should return num_group a series of [0]*5 + [1]*5 + [2]*5 + [3]*5, instead of varying divisions with max(num_group) == e.g. 7

### Show Versions

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.6.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
machine          : AMD64
processor        : Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : None.None

pandas           : 1.0.1
numpy            : 1.15.1
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 20.0.2
setuptools       : 45.2.0.post20200209
Cython           : None
pytest           : 5.3.5
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.11.1
IPython          : 7.12.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : 2.2.3
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pytest           : 5.3.5
pyxlsb           : None
s3fs             : None
scipy            : 1.1.0
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
numba            : None

</details>