Make a Passive Rigid Body Disappear
I have figured out the answer to my problem. It seems you can keyframe a rigid body's collision group and so I did. by pressing I on the collision group layer and positioning your marker to the specific keyframe where you want your rigid body to not affect other meshes.