BUG: to_numeric downcast = 'unsigned' would not un-sign a 0 value.
- [ ] closes #14401
- [ ] tests added / passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry

Have to figure out whats going wrong with the build still. Will let you know as soon as I do.
