How to insulate drywall from odor?
Does this ventilation duct in your closet have an opening into your room?

The most effective method for reducing this odor would be to first identify any air leaks in the shared wall. Insulation will do very little compared to sealing. Leaks are probably the reason you are able to smell the odor to begin with as odors should really not be able to penetrate solid surfaces. Most common sources of air gaps are around electrical outlets, the baseboard where the wall meets the floor, and ventilation outlets. These air gaps can be sealed easily with silicone caulk.

I need a more detailed description of what exactly is going on with this duct in your closet. Is this purely for exhaust/air circulation, heating, or cooling? How easily can you hear noises coming from your neighbor's domicile? There may be serious issues in your home that may be unrelated to your neighbor's living habits. You may have a dead animal or mold hidden somewhere. I absolutely would not do anything you suggested in your initial question. Please provide more details!