[bioformer-8L](https://huggingface.co/bioformers/bioformer-8L) fined-tuned on the [SQuAD1](https://rajpurkar.github.io/SQuAD-explorer) dataset for 3 epochs.

The fine-tuning process was performed on a single P100 GPUs (16GB). The hyperparameters are:

```
max_seq_length=512
per_device_train_batch_size=16
gradient_accumulation_steps=1
total train batch size (w. parallel, distributed & accumulation) = 16
learning_rate=3e-5
num_train_epochs=3
```

## Evaluation results

```
"eval_exact_match": 78.55250709555345
"eval_f1": 85.91482799690257
```

Bioformer's performance is on par with [DistilBERT](https://arxiv.org/pdf/1910.01108.pdf) (EM/F1: 77.7/85.8), 
although Bioformer was pretrained only on biomedical texts. 


## Speed
In our experiments, the inference speed of Bioformer is 3x as fast as BERT-base/BioBERT/PubMedBERT, and is 40% faster than DistilBERT.
{"language": ["en"], "license": "apache-2.0", "pipeline_tag": "question-answering"}
transformers pytorch safetensors bert question-answering en arxiv:1910.01108 license:apache-2.0 endpoints_compatible region:us
TAGS
#transformers #pytorch #safetensors #bert #question-answering #en #arxiv-1910.01108 #license-apache-2.0 #endpoints_compatible #region-us 

bioformer-8L fined-tuned on the SQuAD1 dataset for 3 epochs.

The fine-tuning process was performed on a single P100 GPUs (16GB). The hyperparameters are:



## Evaluation results



Bioformer's performance is on par with DistilBERT (EM/F1: 77.7/85.8), 
although Bioformer was pretrained only on biomedical texts. 


## Speed
In our experiments, the inference speed of Bioformer is 3x as fast as BERT-base/BioBERT/PubMedBERT, and is 40% faster than DistilBERT.
## Evaluation results



Bioformer's performance is on par with DistilBERT (EM/F1: 77.7/85.8), 
although Bioformer was pretrained only on biomedical texts. ## Speed
In our experiments, the inference speed of Bioformer is 3x as fast as BERT-base/BioBERT/PubMedBERT, and is 40% faster than DistilBERT.
TAGS
#transformers #pytorch #safetensors #bert #question-answering #en #arxiv-1910.01108 #license-apache-2.0 #endpoints_compatible #region-us 
 ## Evaluation results



Bioformer's performance is on par with DistilBERT (EM/F1: 77.7/85.8), 
although Bioformer was pretrained only on biomedical texts. ## Speed
In our experiments, the inference speed of Bioformer is 3x as fast as BERT-base/BioBERT/PubMedBERT, and is 40% faster than DistilBERT.
passage: TAGS
#transformers #pytorch #safetensors #bert #question-answering #en #arxiv-1910.01108 #license-apache-2.0 #endpoints_compatible #region-us 
## Evaluation results



Bioformer's performance is on par with DistilBERT (EM/F1: 77.7/85.8), 
although Bioformer was pretrained only on biomedical texts.## Speed
In our experiments, the inference speed of Bioformer is 3x as fast as BERT-base/BioBERT/PubMedBERT, and is 40% faster than DistilBERT.