Check for adjacency after or before using a moving tile?
The answer to your question, as asked, is "The settlement must go back to where you picked it up from."

But your question includes this phrase

I haven't built on any other desert fields yet. So the only neighbouring desert field would be that one.

This is not the correct rule.  Perhaps you meant to say that you haven't built on or next to any other desert fields.  If you've built next to other desert fields, then your settlement could be moved to a desert field next to those settlements.