Sealing existing vinyl tile
Unless there are holes in the vinyl flooring it is probably getting through where the vinyl meets the tub or the wall. Just use a clear or white 100% silicone bath caulk to seal those junctions.

Make sure it is rated for resistance to mold/mildew.