TST: Made s3 related tests mock boto
Kept a couple tests un-mocked  for testing things like accessing a private bucket as that's hard to mock.

- [ ] closes #17325 
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry
