Most people who yearn to stay in shape make an effort to do some form of cardiovascular training three to five times a week for 20 minutes or more per session. With that in mind, anyone looking to swim for fitness should be able to swim at least 20 minutes at a time, several times a week.