was this vent flashing installed totally wrong?
That was installed wrong.  The metal part of the boot should definitely be nailed down.  Since it is not nailed, the wind is free to blow it up.  When it moves up, the rubber seal can become inverted or otherwise deformed.  Repeated movement will end up prematurely cracking the rubber, and of course when the wind blows it up, water will get blown underneath it.

The rubber boot shouldn't need to be caulked if everything is secure.  Normally you'd see a few nails in the metal section, and those nail holes would be caulked.  There should also be a few nails covered up by the course of overlapping shingles.

Also, it's unclear from your photo, but the lower edge of the metal should not extend past the shingle that it is on top of.  if it does, then that row of shingles should have overlapped the metal.  When the metal overhangs the shingle, it gives wind and water more of a chance to lift and penetrate the boot.