Drywall surrounding vent no longer supports screws;
My default fix would be just to put in some type of anchor. Given the damage and the fact this is on a ceiling, I'd be inclined to use a screw-in type anchor, because they hold on to a larger area and would deal better with some damage.



If that couldn't work, the next thing I would look at would be getting a new surface to screw into.

Normally the method is to use a backing board, which is basically just a small piece of wood -- perhaps 3/4" x 2" x 4" -- to screw into on each side. Unless the vent can be moved easily, this will be difficult. 

In this case I'd actually construct some "wings" using some sheet metal to wrap around and over top of the drywall, which the vent would then screw into. They'd anchor into the vent itself.