---
library_name: transformers
tags:
  - biology
  - bd3lm
license: apache-2.0
base_model: kuleshov-group/bd3lm-owt-block_size1024-pretrain
---

# DNA and Block Diffusion

Untrained architecture test using the [Block Diffusion](https://github.com/kuleshov-group/bd3lms) architecture and
[AgroNT](https://huggingface.co/InstaDeepAI/agro-nucleotide-transformer-1b)'s six-nucleotide-length tokens.

### Loading model

```python
from transformers import AutoModelForMaskedLM
m = AutoModelForMaskedLM.from_pretrained(
    "monsoon-nlp/dna-blockdiff",
    trust_remote_code=True,
)
```

### Generating text

```bash
cd bd3lms
python -u main.py \
    loader.eval_batch_size=1 \
    model=small \
    algo=bd3lm \
    algo.T=900 \
    algo.backbone=hf_dit \
    algo.sampler=analytic \
    data=openwebtext-split \
    model.length=2048 \
    block_size=4 \
    wandb=null \
    mode=sample_eval \
    eval.checkpoint_path="monsoon-nlp/dna-blockdiff" \
    model.attn_backend=sdpa \
    sampling.nucleus_p=0.9 \
    sampling.kv_cache=true \
    sampling.logdir=$PWD/sample_logs/samples_genlen_bd3lm_blocksize4 \
    data.tokenizer_name_or_path="monsoon-nlp/dna-blockdiff"
```