Subsequent calls to read_csv for newly created S3 files in the same directory throws an error
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd
import s3fs

BUCKET = 'mattlbeck-mybucket'
s3path1 = f's3://{BUCKET}/key/file1.csv'
s3path2 = f's3://{BUCKET}/key/file2.csv'

with s3fs.S3FileSystem().open(s3path1, 'w') as fh:
    fh.write('a,b,c\n1,2,3')

pd.read_csv(s3path1)

with s3fs.S3FileSystem().open(s3path2, 'w') as fh:
    fh.write('a,b,c\n1,2,3')

pd.read_csv(s3path2)
```

Produces the trace summarised below:

```
Traceback (most recent call last):
  ...
FileNotFoundError: mattlbeck-mybucket/key/file2.csv

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  ...
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 ...
PermissionError: Access Denied
```

#### Problem description

Subsequent calls to `read_csv` beyond the first call causes an error when the target is a file in S3 within the same directory as a file that has previously been read by `read_csv`

The problem appears to be overly aggressive caching of S3 directory listings. But from reading the code I can't immediately see how the cache is persisting between the two calls - there is no obvious reuse of `S3FileSystem` instances.

#### Expected Output

Both calls to `read_csv` should resolve without error because both files exist at the time of the call.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : f20331d543c5d42e54dff29135146fc0e6798a84
python           : 3.8.1.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 19.3.0
Version          : Darwin Kernel Version 19.3.0: Thu Jan  9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : None
LANG             : en_GB.UTF-8
LOCALE           : en_GB.UTF-8

pandas           : 1.1.0.dev0+945.gf20331d54
numpy            : 1.18.2
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 20.0.2
setuptools       : 46.1.1.post20200323
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pyxlsb           : None
s3fs             : 0.4.0
scipy            : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
numba            : None

</details>
