BUG: DataFrame.at setter of categorical DF overwrites entire row
- [x] I have checked that this issue has not already been reported.

- [x] I have confirmed this bug exists on the latest version of pandas.

- [x] (optional) I have confirmed this bug exists on the master branch of pandas.

---

On a DataFrame with categorical dtype, `df.at[x,y] = v` sets all non-initialized values in row `x`.

#### Code Sample, a copy-pastable example

```
$ python
Python 3.8.0 (default, Oct 28 2019, 16:14:01) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
```

```python
>>> import pandas as pd
>>> pd.__version__
'1.2.0.dev0+1137.g50b34a4a8'
>>> df = pd.DataFrame(index=range(3), columns=range(3), dtype=pd.CategoricalDtype(['foo', 'bar']))
>>> df.at[1,2] = 'foo'
>>> df
     0    1    2
0  NaN  NaN  NaN
1  foo  foo  foo
2  NaN  NaN  NaN
```

It doesn't overwrite values that have been set with `df.loc`:

```python
>>> df = pd.DataFrame(index=range(3), columns=range(3), dtype=pd.CategoricalDtype(['foo', 'bar']))
>>> df.loc[1,1] = 'bar'  # not necessary, just for demo
>>> df.at[1,2] = 'foo'
>>> df
     0    1    2
0  NaN  NaN  NaN
1  foo  bar  foo
2  NaN  NaN  NaN
```

#### Problem description

`df.at[x, y] = v` on a categorical dtype should behave as with other dtypes, and the same as `df.loc[x, y] = v`.

#### Expected Output

The same as what happens with a DF initialized with `None`s:

```python
>>> df = pd.DataFrame([[None] * 3] * 3, index=range(3), columns=range(3), dtype=pd.CategoricalDtype(['foo', 'bar']))
>>> df.loc[1,1] = 'bar'
>>> df.at[1,2] = 'foo'
>>> df
     0    1    2
0  NaN  NaN  NaN
1  NaN  bar  foo
2  NaN  NaN  NaN
```

Or as with `dtype=float`:

```python
>>> df = pd.DataFrame(index=range(3), columns=range(3), dtype=float)
>>> df.loc[1,1] = 1
>>> df.at[1,2] = 27
>>> df
    0    1     2
0 NaN  NaN   NaN
1 NaN  1.0  27.0
2 NaN  NaN   NaN
```


#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
<ul>
<li>------------------</li>
<li>commit           : 50b34a4a8008b6da24c3dac6ae6d66536e463239</li>
<li>python           : 3.8.0.final.0</li>
<li>python-bits      : 64</li>
<li>OS               : Linux</li>
<li>OS-release       : 5.4.0-52-generic</li>
<li>Version          : #57~18.04.1-Ubuntu SMP Thu Oct 15 14:04:49 UTC 2020</li>
<li>machine          : x86_64</li>
<li>processor        : x86_64</li>
<li>byteorder        : little</li>
<li>LC_ALL           : None</li>
<li>LANG             : en_US.UTF-8</li>
<li>LOCALE           : en_US.UTF-8</li>
<li></li>
<li>pandas           : 1.2.0.dev0+1137.g50b34a4a8</li>
<li>numpy            : 1.19.4</li>
<li>pytz             : 2020.4</li>
<li>dateutil         : 2.8.1</li>
<li>pip              : 20.2.4</li>
<li>setuptools       : 50.3.2</li>
<li>Cython           : None</li>
<li>pytest           : None</li>
<li>hypothesis       : None</li>
<li>sphinx           : None</li>
<li>blosc            : None</li>
<li>feather          : None</li>
<li>xlsxwriter       : None</li>
<li>lxml.etree       : None</li>
<li>html5lib         : None</li>
<li>pymysql          : None</li>
<li>psycopg2         : None</li>
<li>jinja2           : None</li>
<li>IPython          : None</li>
<li>pandas_datareader: None</li>
<li>bs4              : None</li>
<li>bottleneck       : None</li>
<li>fsspec           : None</li>
<li>fastparquet      : None</li>
<li>gcsfs            : None</li>
<li>matplotlib       : None</li>
<li>numexpr          : None</li>
<li>odfpy            : None</li>
<li>openpyxl         : None</li>
<li>pandas_gbq       : None</li>
<li>pyarrow          : None</li>
<li>pyxlsb           : None</li>
<li>s3fs             : None</li>
<li>scipy            : None</li>
<li>sqlalchemy       : None</li>
<li>tables           : None</li>
<li>tabulate         : None</li>
<li>xarray           : None</li>
<li>xlrd             : None</li>
<li>xlwt             : None</li>
<li>numba            : None</li>
</ul>
</details>
