
# sgpt-bloom-1b7-nli

## Usage

For usage instructions, refer to: https://github.com/Muennighoff/sgpt#symmetric-semantic-search

The model was trained with the command
```bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 accelerate launch examples/training/nli/training_nli_v2.py --model_name bigscience/bloom-1b3 --freezenonbias --train_batch_size 128 --lr 32e-5 --pooling weightedmean --wandb --wandbwatchlog gradients --gradcache --chunksize 4
```

## Evaluation Results

`{'askubuntu': 57.44, 'cqadupstack': 14.18, 'twitterpara': 73.99, 'scidocs': 74.74, 'avg': 55.087500000000006}`

## Training
The model was trained with the parameters:

**DataLoader**:

`sentence_transformers.datasets.NoDuplicatesDataLoader.NoDuplicatesDataLoader` of length 4403 with parameters:
```
{'batch_size': 128}
```

The model uses BitFit, weighted-mean pooling & GradCache, for details see: https://arxiv.org/abs/2202.08904

**Loss**:

`sentence_transformers.losses.MultipleNegativesRankingLoss.MNRLGradCache` 

Parameters of the fit()-Method:
```
{
    "epochs": 1,
    "evaluation_steps": 440,
    "evaluator": "sentence_transformers.evaluation.EmbeddingSimilarityEvaluator.EmbeddingSimilarityEvaluator",
    "max_grad_norm": 1,
    "optimizer_class": "<class 'transformers.optimization.AdamW'>",
    "optimizer_params": {
        "lr": 0.00032
    },
    "scheduler": "WarmupLinear",
    "steps_per_epoch": null,
    "warmup_steps": 441,
    "weight_decay": 0.01
}
```


## Full Model Architecture
```
SentenceTransformer(
  (0): Transformer({'max_seq_length': 75, 'do_lower_case': False}) with Transformer model: BloomModel 
  (1): Pooling({'word_embedding_dimension': 2048, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': True, 'pooling_mode_lasttoken': False})
)
```

## Citing & Authors

```bibtex
@article{muennighoff2022sgpt,
  title={SGPT: GPT Sentence Embeddings for Semantic Search},
  author={Muennighoff, Niklas},
  journal={arXiv preprint arXiv:2202.08904},
  year={2022}
}
```

{"tags": ["sentence-transformers", "feature-extraction", "sentence-similarity", "mteb"], "pipeline_tag": "sentence-similarity", "model-index": [{"name": "sgpt-bloom-1b7-nli", "results": [{"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (fr)", "type": "mteb/amazon_reviews_multi", "config": "fr", "split": "test", "revision": "c379a6705fec24a2493fa68e011692605f44e119"}, "metrics": [{"type": "accuracy", "value": 39.286}, {"type": "f1", "value": 38.87078070073539}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (zh)", "type": "mteb/amazon_reviews_multi", "config": "zh", "split": "test", "revision": "c379a6705fec24a2493fa68e011692605f44e119"}, "metrics": [{"type": "accuracy", "value": 37.634}, {"type": "f1", "value": 36.86046604093418}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPDomainClassification (fr)", "type": "mteb/mtop_domain", "config": "fr", "split": "test", "revision": "a7e2a951126a26fc8c6a69f835f33a346ba259e3"}, "metrics": [{"type": "accuracy", "value": 83.79893517068588}, {"type": "f1", "value": 83.72326662566203}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPIntentClassification (fr)", "type": "mteb/mtop_intent", "config": "fr", "split": "test", "revision": "6299947a7777084cc2d4b64235bf7190381ce755"}, "metrics": [{"type": "accuracy", "value": 63.36047604134043}, {"type": "f1", "value": 44.261707019308126}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (fr)", "type": "mteb/amazon_massive_intent", "config": "fr", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 64.57632817753867}, {"type": "f1", "value": 62.60453982786661}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (fr)", "type": "mteb/amazon_massive_scenario", "config": "fr", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 69.59986550100874}, {"type": "f1", "value": 69.71803697939914}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (zh)", "type": "mteb/sts22-crosslingual-sts", "config": "zh", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 59.71781185663265}, {"type": "cos_sim_spearman", "value": 58.538648447630514}, {"type": "euclidean_pearson", "value": 53.53848180206165}, {"type": "euclidean_spearman", "value": 56.33730262964236}, {"type": "manhattan_pearson", "value": 54.62109820575505}, {"type": "manhattan_spearman", "value": 57.223846291318914}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (fr)", "type": "mteb/sts22-crosslingual-sts", "config": "fr", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 73.44021434651606}, {"type": "cos_sim_spearman", "value": 73.13412769502769}, {"type": "euclidean_pearson", "value": 68.16368597409867}, {"type": "euclidean_spearman", "value": 72.44964781564485}, {"type": "manhattan_pearson", "value": 69.42307032478939}, {"type": "manhattan_spearman", "value": 73.3523195012387}]}]}]}
sentence-transformers pytorch bloom feature-extraction sentence-similarity mteb arxiv:2202.08904 model-index endpoints_compatible has_space region:us
TAGS
#sentence-transformers #pytorch #bloom #feature-extraction #sentence-similarity #mteb #arxiv-2202.08904 #model-index #endpoints_compatible #has_space #region-us 


# sgpt-bloom-1b7-nli

## Usage

For usage instructions, refer to: URL

The model was trained with the command


## Evaluation Results

'{'askubuntu': 57.44, 'cqadupstack': 14.18, 'twitterpara': 73.99, 'scidocs': 74.74, 'avg': 55.087500000000006}'

## Training
The model was trained with the parameters:

DataLoader:

'sentence_transformers.datasets.NoDuplicatesDataLoader.NoDuplicatesDataLoader' of length 4403 with parameters:


The model uses BitFit, weighted-mean pooling & GradCache, for details see: URL

Loss:

'sentence_transformers.losses.MultipleNegativesRankingLoss.MNRLGradCache' 

Parameters of the fit()-Method:



## Full Model Architecture


## Citing & Authors



# sgpt-bloom-1b7-nli ## Usage

For usage instructions, refer to: URL

The model was trained with the command ## Evaluation Results

'{'askubuntu': 57.44, 'cqadupstack': 14.18, 'twitterpara': 73.99, 'scidocs': 74.74, 'avg': 55.087500000000006}' ## Training
The model was trained with the parameters:

DataLoader:

'sentence_transformers.datasets.NoDuplicatesDataLoader.NoDuplicatesDataLoader' of length 4403 with parameters:


The model uses BitFit, weighted-mean pooling & GradCache, for details see: URL

Loss:

'sentence_transformers.losses.MultipleNegativesRankingLoss.MNRLGradCache' 

Parameters of the fit()-Method: ## Full Model Architecture ## Citing & Authors
TAGS
#sentence-transformers #pytorch #bloom #feature-extraction #sentence-similarity #mteb #arxiv-2202.08904 #model-index #endpoints_compatible #has_space #region-us 
 # sgpt-bloom-1b7-nli ## Usage

For usage instructions, refer to: URL

The model was trained with the command ## Evaluation Results

'{'askubuntu': 57.44, 'cqadupstack': 14.18, 'twitterpara': 73.99, 'scidocs': 74.74, 'avg': 55.087500000000006}' ## Training
The model was trained with the parameters:

DataLoader:

'sentence_transformers.datasets.NoDuplicatesDataLoader.NoDuplicatesDataLoader' of length 4403 with parameters:


The model uses BitFit, weighted-mean pooling & GradCache, for details see: URL

Loss:

'sentence_transformers.losses.MultipleNegativesRankingLoss.MNRLGradCache' 

Parameters of the fit()-Method: ## Full Model Architecture ## Citing & Authors
passage: TAGS
#sentence-transformers #pytorch #bloom #feature-extraction #sentence-similarity #mteb #arxiv-2202.08904 #model-index #endpoints_compatible #has_space #region-us 
# sgpt-bloom-1b7-nli## Usage

For usage instructions, refer to: URL

The model was trained with the command## Evaluation Results

'{'askubuntu': 57.44, 'cqadupstack': 14.18, 'twitterpara': 73.99, 'scidocs': 74.74, 'avg': 55.087500000000006}'## Training
The model was trained with the parameters:

DataLoader:

'sentence_transformers.datasets.NoDuplicatesDataLoader.NoDuplicatesDataLoader' of length 4403 with parameters:


The model uses BitFit, weighted-mean pooling & GradCache, for details see: URL

Loss:

'sentence_transformers.losses.MultipleNegativesRankingLoss.MNRLGradCache' 

Parameters of the fit()-Method:## Full Model Architecture## Citing & Authors