
# CompoundT5 

This model is a re-pretrained version of [google/t5-v1_1-base](https://huggingface.co/microsoft/deberta-base) on the sagawa/ZINC-canonicalized dataset.
It achieves the following results on the evaluation set:
- Loss: 0.1202
- Accuracy: 0.9497


## Model description

We trained t5 on SMILES from ZINC using masked-language modeling (MLM). Its tokenizer is also trained on ZINC.


## Intended uses & limitations

This model can be used to predict molecules' properties, reactions, or interactions with proteins by changing the way of finetuning.
As an example, We finetuned this model to predict products. The model is [here](https://huggingface.co/sagawa/ZINC-t5-productpredicition), and you can use the demo [here](https://huggingface.co/spaces/sagawa/predictproduct-t5).
Using its encoder, we trained a regression model to predict a reaction yield. You can use this demo [here](https://huggingface.co/spaces/sagawa/predictyield-t5).

## Training and evaluation data

We downloaded [ZINC data](https://drive.google.com/drive/folders/1lSPCqh31zxTVEhuiPde7W3rZG8kPgp-z) and canonicalized them using RDKit. Then, we dropped duplicates. The total number of data is 22992522, and they were randomly split into train:validation=10:1.

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-03
- train_batch_size: 30
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 30.0

### Training results

| Training Loss | Step   | Accuracy | Validation Loss |
|:-------------:|:------:|:--------:|:---------------:|
| 0.2471        | 25000  | 0.9843   | 0.2226          |
| 0.1871        | 50000  | 0.9314   | 0.1783          |
| 0.1791        | 75000  | 0.9371   | 0.1619          |
| 0.1596        | 100000 | 0.9401   | 0.1520          |
| 0.1522        | 125000 | 0.9422   | 0.1449          |
| 0.1435        | 150000 | 0.9436   | 0.1404          |
| 0.1421        | 175000 | 0.9447   | 0.1368          |
| 0.1398        | 200000 | 0.9459   | 0.1322          |
| 0.1297        | 225000 | 0.9466   | 0.1299          |
| 0.1324        | 250000 | 0.9473   | 0.1268          |
| 0.1257        | 275000 | 0.9483   | 0.1244          |
| 0.1266        | 300000 | 0.9491   | 0.1216          |
| 0.1301        | 325000 | 0.9497   | 0.1204          |
{"license": "mit", "datasets": ["sagawa/ZINC-canonicalized"], "metrics": ["accuracy"], "model-index": [{"name": "ZINC-deberta", "results": [{"task": {"type": "fill-mask", "name": "Masked Language Modeling"}, "dataset": {"name": "sagawa/ZINC-canonicalized", "type": "sagawa/ZINC-canonicalized"}, "metrics": [{"type": "accuracy", "value": 0.9497212171554565, "name": "Accuracy"}]}]}]}
transformers pytorch jax t5 text2text-generation dataset:sagawa/ZINC-canonicalized license:mit model-index autotrain_compatible endpoints_compatible text-generation-inference region:us
TAGS
#transformers #pytorch #jax #t5 #text2text-generation #dataset-sagawa/ZINC-canonicalized #license-mit #model-index #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 

CompoundT5
==========


This model is a re-pretrained version of google/t5-v1\_1-base on the sagawa/ZINC-canonicalized dataset.
It achieves the following results on the evaluation set:


* Loss: 0.1202
* Accuracy: 0.9497


Model description
-----------------


We trained t5 on SMILES from ZINC using masked-language modeling (MLM). Its tokenizer is also trained on ZINC.


Intended uses & limitations
---------------------------


This model can be used to predict molecules' properties, reactions, or interactions with proteins by changing the way of finetuning.
As an example, We finetuned this model to predict products. The model is here, and you can use the demo here.
Using its encoder, we trained a regression model to predict a reaction yield. You can use this demo here.


Training and evaluation data
----------------------------


We downloaded ZINC data and canonicalized them using RDKit. Then, we dropped duplicates. The total number of data is 22992522, and they were randomly split into train:validation=10:1.


Training procedure
------------------


### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 5e-03
* train\_batch\_size: 30
* eval\_batch\_size: 32
* seed: 42
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* num\_epochs: 30.0


### Training results




### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 5e-03
* train\_batch\_size: 30
* eval\_batch\_size: 32
* seed: 42
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* num\_epochs: 30.0 ### Training results
TAGS
#transformers #pytorch #jax #t5 #text2text-generation #dataset-sagawa/ZINC-canonicalized #license-mit #model-index #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
 ### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 5e-03
* train\_batch\_size: 30
* eval\_batch\_size: 32
* seed: 42
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* num\_epochs: 30.0 ### Training results
passage: TAGS
#transformers #pytorch #jax #t5 #text2text-generation #dataset-sagawa/ZINC-canonicalized #license-mit #model-index #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 5e-03
* train\_batch\_size: 30
* eval\_batch\_size: 32
* seed: 42
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* num\_epochs: 30.0### Training results