ENH/CI: Cache packages installed from conda
Currently, I think we are downloading all the packages every time CI is run. This takes a long time(~5-10 min. on azure) and makes CI flakier, since sometimes we can't connect to the Anaconda servers, and sometimes we timeout on Azure. We should cache the packages downloaded and only check for updates everytime the CI is run. I think we're doing some form of this on GHA, but we should also do it on azure.

cc @fangchenli 
