TST: Add test cases for GH6173, appending to empty df
- [X] closes #6173
- [X] tests added / passed
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Hmm, previous check-in for GH4861 still showing in this branch, thought it wouldn't show as a diff after syncing to upstream (sorry, new to git)