Ester foam insert and isopropyl alcohol
I would just try soap and water first. If you want to try any kind of solvent like alcohols, you really needn't be concerned with lingering smell, but I would be concerned with the (slight) possibility of destroying the foam. 

If soap and water doesn't work, just try placing a drop of isopropanol on both the top and bottom of the insert and let it dry. If there is any change in the consistency or feel to it I would stop there. If not, then progress to cleaning a small area, and if that does no damage go ahead and soak the whole thing if needed.