I have an exterior water tap for a garden hose that needs new caulking
The best I can suggest is remove that old wood,it may attract insects as well,I used closed cell spray foam to seal my numerous entry holes,and the closed cell should help stabilize the pipe.If you have acess you could stabilize it with a bracket or hanger if its in the crawl space or basement.Good luck