How to compare two unsupervised anomaly detection algorithms on the same data-set?
For unlabeled data-sets, unsupervised anomaly detectors can be compared either subjectively or objectively.


Subjective comparison: based on our domain-knowledge and by using some visualizations and statistics, we can compare two detectors and select the one that outputs better anomalies subjectively.


Here is a well-cited survey on unsupervised anomaly detectors that compares the algorithms on labeled data-sets (with known, domain-specific outliers) using AUC, and concludes that local detectors (such as LOF,
COF, INFLO and LoOP) are not good candidates for global anomaly detection: 
2016 A Comparative Evaluation of Unsupervised Anomaly Detection Algorithms for Multivariate Data

Objective comparison: possible in theory, impossible in practice.


Requirements for objective comparison:


Anomaly definition: $x$ is an anomaly if $P(x)&lt; t$ for some threshold $t$,
Anomaly detector requirement: $D$ is an anomaly detector if for every detected $x$, $P(x)&lt; t$,
Comparing anomalies: $x_1$ is more anomalous than $x_2$ if $P(x_1)&lt;P(x_2)$ or equivalently $r(x_1, x_2) = P(x_1) / P(x_2) &lt; 1$,
Comparing anomaly detectors: proposal $x_1$ from detector $D_1$ is better than $x_2$ from $D_2$ if $r(x_1, x_2) &lt; 1$,


As you can see, for qualification and comparison of two detectors we need to know the underlying $P(x)$ or at least $r(x_1, x_2)$. But if we know these quantities (which act as a  judge $J$) or at least a close enough estimation of them, we have a better anomaly detector $J$ and can throw $D_1$ and $D_2$ away! We plug any observation $x$ or pair of observations $x_1$ and $x_2$ into $J$ and check which one is an anomaly or which one is more anomalous, done! So it is impossible to compare two anomaly detectors objectively unless we have a better anomaly detector (judge). So we should use a subjective comparison.