What is the significance of enrichment factor regarding machine learning methods? I'm fairly new to the field of bioinformatics and ran into a question while reading a paper I found on bioRxiv. The overall setting of the paper is using multi-task deep neural networks for kinase classification. The paper that I'm referring to is Kinome-Wide Activity Classification of Small Molecules by Deep Learning (Allen et al., 2019) for anyone who's curious.

I'll get straight into the specific figure that I'm having trouble understanding.



I know that questions like these are a bit discouraged on SE, but I'm having trouble on how to start regarding the interpretation of this figure.

Based on my understanding, the point seems to be that multitaks DNN's have the highest overall EF/EFmax ratio amounts, but what does this ratio signify?

I did some research and according to Wikipedia an "enrichment factor" in terms of bioinformatics is a concept "to measure the added value of a search tool over another one or over the homogeneous distribution in the genome population." Does this mean that multitask DNN's are "better" than the other models, due to its higher number of EF/EFmax points?

But if we look at the curves on the right, it seems to say that for fewer number of compounds, the ratio for multitask DNN's is lower...
I did some research and according to Wikipedia an "enrichment factor" in terms of bioinformatics is...


Where possible, it's best to use explanations provided in the original source. This is because definitions could be different, depending on who is writing. Here's some descriptive information from the paper that may be more informative:


  The enrichment factor (EF) is another metric we use and is very
  popular for use in evaluation of virtual screening performance. EF
  denotes the quotient of true actives among a subset of predicted
  actives and the overall fraction of actives.... To evaluate the
  maximum achievable enrichment factor (EFmax), the maximum number of
  actives among the percentage of tested compounds is divided by the
  fraction of actives in the entire dataset. To more consistently
  quantify enrichment at 0.1% of all compounds tested, we report the
  ratio of EF/ EFmax. This normalized metric is useful because EF values
  are not directly comparable across different datasets due to the
  maximum possible EF being constrained by the ratio of total to active
  compounds and the evaluated fraction as shown in the equation above.


So for the purpose of this paper, they're saying that this ratio is an appropriately normalised substitute for a plain enrichment factor. High enrichment factor == good; low enrichment factor == bad.

Given the information you've provided, I'd agree that the curves in this figure appear to suggest that the multitask DNN performs worse with smaller numbers of active compounds (where Naive Bayes or Logistic regression would be a better choice). It's hard to tell from the graph (and legend) what the curves represent, and the clipping at EF/EFmax = 0 doesn't help.

The curves have a strange shape, suggesting the number of active compounds is dependent on the EF/EFmax ratio, rather than the other way round (which would make more sense to me). If I consider the curves to be something like a smoothed mean of the active compound number, then any place where there are fewer dots (i.e. low EF/EFmax for Multitask DNN) will have more associated error. For Multitask DNN, I see a large gap between 0.0 and 0.2, with lots of outliers on the 0.0 line (for all methods), so presumably the 0.0 outliers are skewing the curves.

Given that this is a preprint, the generation of the curves isn't specified, and the other figures (especially Figure 3) seem to tell a different story (i.e. that multi-task DNN performs better at all scales), I'd recommend contacting the authors to ask / clarify what the curves represent, or ignoring the curves entirely.