You've said the best way to come up with chords is by selecting the chord(s) that best support the melody... But, what about if you're accompanying a soloist who is improvising and you don't know what the melody notes are going to be? How would you come up with interesting chord substitutions/changes to support the melody if you don't know where the melody is going?
start by doing the most basic thing according to the song form. then as you go, you should get a feel for where they are going, and they will get a feel for where you are going. it gets better and better with experience. don't forget to have fun, stretch out, and relax kind of like you are in a lounge