Is it much more difficult to enter an Ivy League much later in life?
Regular college admissions are mainly set up to handle people just graduating from high school. But they do get applications from older students and this can often be beneficial in promoting a certain type of diversity in their incoming class.

Many universities also have explicit admissions and scholarship programs for older students. These often free you from the requirement to live in freshman dorms with 18 year olds or allow you to study part-time.

This is just one example: http://admissions.yale.edu/eli-whitney