philipphager
/

baidu-ultr_uva-bert_ips-pointwise

Inference Endpoints

Model card Files Files and versions Community

philipphager commited on Apr 30, 2024

Commit

d87bfc4

·

verified ·

1 Parent(s): afe7a44

Update README.md

Files changed (1) hide show

README.md +10 -2

README.md CHANGED Viewed

@@ -16,9 +16,17 @@ metrics:
 # Pointwise MonoBERT trained on Baidu-ULTR with Inverse Propensity Scoring (IPS)
 A flax-based MonoBERT cross encoder trained on the [Baidu-ULTR](https://arxiv.org/abs/2207.03051) dataset with the **pointwise sigmoid cross-entropy loss with IPS correction** suggested by [Bekker et al.](https://arxiv.org/abs/1809.03207) and [Saito et al.](https://arxiv.org/abs/1909.03601). The loss uses inverse propensity scoring to mitigate position bias in click data by weighting clicks on items higher that are less likely to be observed by users. For more info, [read our paper](https://arxiv.org/abs/2404.02543) and [find the code for this model here](https://github.com/philipphager/baidu-bert-model).
-## Test Results on Baidu-ULTR Expert Annotations
 ## Usage

 # Pointwise MonoBERT trained on Baidu-ULTR with Inverse Propensity Scoring (IPS)
 A flax-based MonoBERT cross encoder trained on the [Baidu-ULTR](https://arxiv.org/abs/2207.03051) dataset with the **pointwise sigmoid cross-entropy loss with IPS correction** suggested by [Bekker et al.](https://arxiv.org/abs/1809.03207) and [Saito et al.](https://arxiv.org/abs/1909.03601). The loss uses inverse propensity scoring to mitigate position bias in click data by weighting clicks on items higher that are less likely to be observed by users. For more info, [read our paper](https://arxiv.org/abs/2404.02543) and [find the code for this model here](https://github.com/philipphager/baidu-bert-model).
+## Test Results on Baidu-ULTR
+Ranking performance is measured in DCG, nDCG, and MRR on expert annotations (6,985 queries). Click prediction performance is measured in log-likelihood on one test partition of user clicks (49,495 queries).
+| Model                                                                                          | Log-likelihood | DCG@1 | DCG@3 | DCG@5 | DCG@10 | nDCG@10 | MRR@10 |
+|------------------------------------------------------------------------------------------------|----------------|-------|-------|-------|--------|---------|--------|
+| [Pointwise Naive](https://huggingface.co/philipphager/baidu-ultr_uva-bert_naive-pointwise)     | 0.227          | 1.641 | 3.462 | 4.752 | 7.251  | 0.357   | 0.609  |
+| [Pointwise Two-Tower](https://huggingface.co/philipphager/baidu-ultr_uva-bert_twotower)        | 0.218          | 1.629 | 3.471 | 4.822 | 7.456  | 0.367   | 0.607  |
+| [Pointwise IPS](https://huggingface.co/philipphager/baidu-ultr_uva-bert_ips-pointwise)         | 0.222          | 1.295 | 2.811 | 3.977 | 6.296  | 0.307   | 0.534  |
+| [Listwise Naive](https://huggingface.co/philipphager/baidu-ultr_uva-bert_naive-listwise)       | -              | 1.947 | 4.108 | 5.614 | 8.478  | 0.405   | 0.639  |
+| [Listwise IPS](https://huggingface.co/philipphager/baidu-ultr_uva-bert_ips-listwise)           | -              | 1.671 | 3.530 | 4.873 | 7.450  | 0.361   | 0.603  |
+| [Listwise DLA](https://huggingface.co/philipphager/baidu-ultr_uva-bert_dla)                    | -              | 1.796 | 3.730 | 5.125 | 7.802  | 0.377   | 0.615  |
 ## Usage