What is "GOAL" in terms of Reinforcement Learning specified in these papers?
After some discussions with Kiuhnm in RL Group Discord channel, I understand it now.
In the Horde paper it is easy, i.e. the goal might be state or a time. This sub-agent (called demon in the paper) might have a goal of running for 10 time steps, and you might have a reward function specific for it.

In the UVFA paper, the example was to define goals as a state and that's just for simplicity. Let's say you are in a room and your state is 2-dimensional position. Your goal maybe going to next room, and that means another 2-dimensional position. UVFA proposes that, the network will generalize over goals, and make your goal achieving easier in the following tasks.

That pretty much sums it up. Thanks to everyone who did give a thought.