|
--- |
|
library_name: transformers |
|
tags: |
|
- biology |
|
- bd3lm |
|
license: apache-2.0 |
|
base_model: kuleshov-group/bd3lm-owt-block_size1024-pretrain |
|
--- |
|
|
|
# DNA and Block Diffusion |
|
|
|
Untrained architecture test using the [Block Diffusion](https://github.com/kuleshov-group/bd3lms) architecture and |
|
[AgroNT](https://huggingface.co/InstaDeepAI/agro-nucleotide-transformer-1b)'s six-nucleotide-length tokens. |
|
|
|
### Loading model |
|
|
|
```python |
|
from transformers import AutoModelForMaskedLM |
|
m = AutoModelForMaskedLM.from_pretrained( |
|
"monsoon-nlp/dna-blockdiff", |
|
trust_remote_code=True, |
|
) |
|
``` |
|
|
|
### Generating text |
|
|
|
```bash |
|
cd bd3lms |
|
python -u main.py \ |
|
loader.eval_batch_size=1 \ |
|
model=small \ |
|
algo=bd3lm \ |
|
algo.T=900 \ |
|
algo.backbone=hf_dit \ |
|
algo.sampler=analytic \ |
|
data=openwebtext-split \ |
|
model.length=2048 \ |
|
block_size=4 \ |
|
wandb=null \ |
|
mode=sample_eval \ |
|
eval.checkpoint_path="monsoon-nlp/dna-blockdiff" \ |
|
model.attn_backend=sdpa \ |
|
sampling.nucleus_p=0.9 \ |
|
sampling.kv_cache=true \ |
|
sampling.logdir=$PWD/sample_logs/samples_genlen_bd3lm_blocksize4 \ |
|
data.tokenizer_name_or_path="monsoon-nlp/dna-blockdiff" |
|
``` |
|
|
