CI: Fixing possible bugs in the CI
I'm not sure if they are actual bugs, or I'm not understanding something. I think it's easier to discuss over a PR, so I fixed what I think it's wrong.

@jreback @TomAugspurger do you mind taking a look?