How to seal dryer exhaust vent
In the end I used a sealant putty called Mortite to fill in the space between the tube and the brick. I tried low expansion foam but as it expanded and dried, it squeezed the tube out of shape so that I couldn't fit the cap back on. There was really no way to caulk that area because once the cap is on, you can no longer access it.
So basically, I put the vent into the hole with the cap on, secured it to the wall on the other side to it wouldn't move, and took the cap off. Then I smooshed the putty into the cracks and placed the cap back on. Had to get some help to make it work but it wasn't too bad overall.
I also went ahead and used the putty to seal the metal collar to the wall on the other side where the dryer hose connects.
Also I used Loctite to glue a piece of broken stone back onto the wall on the right of the hole.