Pandas: cannot set a Timestamp with a non-timestamp

I have a data-frame of the following order: https://imgur.com/a/fp2hKZe on which i ran the following code 
```python
for i in r.index[1:]:
    if (r.at[i, 'lbl'] == r.at[i-1, 'lbl']) & (r.at[i, 'lbl'] == 'f'):
        r.at[i, 'date'] = r.at[i, 'date'] - r.at[i-1, 'date']
    elif r.at[i, 'lbl'] == 'f':
        r.at[i, 'date'] = r.at[i, 'date'] - r.at[i-1, 'date']
``` 
#### Problem description

When i run the same code for a much smaller data-set, it gives proper output
```python
data = pd.DataFrame(data=[[101,'f',datetime.date(2017,12,28)],
                         [101, 'f', datetime.date(2019,8,27)],
                         [101,'m',datetime.date(2013,4,12)],
                         [101,'m',datetime.date(2015,6,25)],
                         [101, 'f', datetime.date(2016,8,16)],
                         [102,'m', datetime.date(2019,8,15)],
                         [102,'m',datetime.date(2019,6,25)]],
                  columns=["TC", "lbl", "date"])
```


```python
for i in data.index[1:]:
    if (data.at[i, 'lbl'] == data.at[i-1, 'lbl']) & (data.at[i, 'lbl'] == 'f'):
        data.at[i, 'date'] = data.at[i, 'date'] - data.at[i-1, 'date']
    elif data.at[i, 'lbl'] == 'f':
        data.at[i, 'date'] = data.at[i, 'date'] - data.at[i-1, 'date']
```

But when i change the  data-set, i get The **ValueError**
#### Expected Output

  | TC | lbl | date
-- | -- | -- | --
0 | 101 | f | 2017-12-28
1 | 101 | f | 607 days, 0:00:00
2 | 101 | m | 2013-04-12
3 | 101 | m | 2015-06-25
4 | 101 | f | 418 days, 0:00:00
5 | 102 | m | 2019-08-15
6 | 102 | m | 2019-06-25


#### Output of ``pd.show_versions()``

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.4.final.0
python-bits      : 64
OS               : Linux
OS-release       : 5.0.0-32-generic
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_IN
LOCALE           : en_IN.ISO8859-1

pandas           : 0.25.1
numpy            : 1.17.2
pytz             : 2019.3
dateutil         : 2.8.0
pip              : 19.2.3
setuptools       : 41.4.0
Cython           : 0.29.13
pytest           : 5.2.1
hypothesis       : None
sphinx           : 2.2.0
blosc            : None
feather          : None
xlsxwriter       : 1.2.1
lxml.etree       : 4.4.1
html5lib         : 1.0.1
pymysql          : None
psycopg2         : None
jinja2           : 2.10.3
IPython          : 7.8.0
pandas_datareader: None
bs4              : 4.8.0
bottleneck       : 1.2.1
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.4.1
matplotlib       : 3.1.1
numexpr          : 2.7.0
odfpy            : None
openpyxl         : 3.0.0
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : 1.3.1
sqlalchemy       : 1.3.9
tables           : 3.5.2
xarray           : None
xlrd             : 1.2.0
xlwt             : 1.3.0
xlsxwriter       : 1.2.1

