How is "relevance" defined in information retrieval outside the context of systems with user feedback?
There is no formal definition for the concept of relevance, because relevance depends completely on the context and is therefore highly subjective. This is why the best way (some might say the only way) to evaluate relevance is to actually ask users what is relevant for them.

For any ML-based task, one needs to design a proper evaluation framework in order to  control and measure the quality of the results. Naturally the evaluation method should be chosen so that it reflects as much as possible the level of quality with respect to the goal of the task, i.e. what one would intuitively expect from it. Evaluation metrics are almost always simplified indicators of this "level of quality", so what matters is how well they correlate with what a user would expect from the system: sometimes even a perfectly standard evaluation measure might not to be suited to the goal of the task.

My point is that evaluation is a matter of analysis and design. There are infinitely many options, but the point is to select the most appropriate one for the job. Here are some of these options:


The ideal case is to have annotated data (e.g. user feedback) directly suited to the data and the task: then it's just a matter of counting how often the prediction is correct.
It's common to evaluate a system against another annotated dataset X, assuming that the task is similar enough so if the system works well on X then it will work well on the real dataset.
Another less than ideal way is to evaluate against the predictions of another reference system X: in this case X is considered the gold standard, so there is no way for the tested system to perform better than X.
Indirect evaluation: if there is another task being performed at a later stage with the predictions, and this task can be evaluated more easily than the IR task itself.
Heuristics: that would be the least reliable kind of evaluation, but it's better than nothing. It ranges from simply counting the number of words in common between the query and top N results to developing complex methods using third-party resources.