Post-tenure move: Reference letter from institution to which I'm applying
A letter written by someone the reader knows is likely to be more credible than a letter written by someone unknown.  So it makes sense to get a letter from someone at X when you know the reader will be at X.

they might be seen as having a vested interest

Most letter writers have an interest in the success of the person they are recommending.  A writer who works at X will have additional knowledge of the interests of the whole university, which could result in a more useful letter.

would it be weird to draw on a reference from this collaborator when applying to the University of X?

No.