Structural damage because of air vent
So long as you're not removing a stud or part of a stud, you should be fine. Just punching a 4" hole through sheetrock, insulation, and the outside covering (brick, vinyl, aluminum, etc) will not compromise the structural integrity of the wall.