How can certain information about the goal be given to the RL learning algorithm?
I see three main ways to do this, which one makes more sense will depend on your application.

One is to append that information to the state/observations like you mentioned. While this information is static for a particular episode, it will be different across episodes and the policy should learn to condition the actions it chooses based on this information.

Another would be to leave goal information out entirely and force the agent to learn a policy that works when the goal is unknown. This will likely be more difficult to learn and you may end up with a policy that moves to the average of all goals or explores and tries them all.

A third option and probably the most natural is to have some context cue that the agent can observe (e.g. often in lab experiments with rats, a cue is placed on a wall that lets the rat know which way to go to get a reward). This is similar to the first method, except that the cue has to be observed rather than given directly. For the Mountain Car example, this could be an extra signal that the agent only sees in a particular location, such as the bottom of the valley or when it moves close to a particular side.