Is the target the source of the difficulty?
So basically, am I right in saying that the difficulty is calculated from the target, and not the other way around?


That is correct. Difficulty is just a concept introduced to make the target value easy to represent for humans.  It has no meaning internally.