Why does my bath fan duct have dried grass in it?
Yes, you have a rodent or bird nest in there. No, you shouldn't break anything. 

That vent flap is simply hanging on two little integral pins or arms. You can see them in the photo. You should be able to tilt the flap into alignment with the duct, then flex it to release the arm on one side. 

Do your cleaning and replace the flap. Now find out why your duct is open to the outside. It should probably have mesh over the opening to prevent pest intrusion.