When your 4-6 month old baby is learning to use a cup, giving him a few sips of water a couple of times a day (no more than 2 ounces per 24 hours) is fine and fun. Once baby starts solids, you might want to give him a few sips of expressed milk or water with his solids – some babies need this to prevent constipation.