What's the difference between a No.4 and No.5 hand plane
The plane iron may also be ground slight differently in shape (as opposed to angle)... but yes, those are the largest differences. They do affect how the plane is used.

A longer plane improves its ability to accurately flatten a surface (it can level out wider-spaced hills and valleys where a shorter plane would just follow the curves). But depending on what you're doing, the additional weight may be tiring or may be helpful -- harder to lift, but the weight helps keep the iron in contact with the wood.

The ideal hand-tool woodshop has a range of planes from jointers (long) all the way down to block planes small enough to hold in a hand, plus various specialty planes (shoulder planes, rabbeting planes, router planes...) each of them having its own trade-offs -- just as the ideal woodshop has a range of chisel sizes and types. Most of us don't have ideal shops, and that's OK. You can do the same jobs with a smaller setup, it may just take more work and care.

If I had to pick just one of those two, I think I'd have gone for the #4 first... but that may be personal bias; I've already rescued several of that size from various garage sales, and because I haven't had to do serious smoothing/jointing yet. And I'd suggest a block plane as your second purchase -- they're light enough to carry anywhere and more useful for spot-trimming than you might think.

If you do buy a 9" plane, it's seriously worth considering a low-angle plane (lower bedding angle, higher angle on the blade). Among other things, that gives you a wider range available if you need to grind blades to other angles when handling special cases. These do tend to be a bit more expensive, though, largely because this design is mostly offered by the higher-end manufacturers.

(Edit: Yes, I know, "long jointers are long, duh". Redundancy fixed.)