
# wav2vec2-base-cs-80k-ClTRUS
**C**zech **l**anguage **TR**ransformer from **U**nlabeled **S**peech (ClTRUS) is a monolingual Czech Wav2Vec 2.0 base model pre-trained from 80 thousand hours of Czech speech.

This model does not have a tokenizer as it was pretrained on audio alone. In order to use this model for speech recognition, a tokenizer should be created and the model should be fine-tuned on labeled text data. 

**Note:** This is a checkpoint of the model after 4 epochs over the whole dataset. If you want some earlier or later checkpoints, please feel free to contact the author (jlehecka(at)kky.zcu.cz).

## Pretraining data 
More than 80 thousand hours of unlabeled Czech speech:
- recordings from radio (22k hours),
- unlabeled data from VoxPopuli dataset (18.7k hours),
- TV shows (15k hours), 
- shadow speakers (12k hours), 
- sports (5k hours), 
- telephone data (2k hours),
- and a smaller amount of data from several other domains including the public CommonVoice dataset.

## Usage
Inputs must be 16kHz mono audio files.

This model can be used e.g. to extract per-frame contextual embeddings from audio:
```python
from transformers import Wav2Vec2Model, Wav2Vec2FeatureExtractor
import torchaudio

feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("fav-kky/wav2vec2-base-cs-80k-ClTRUS")
model = Wav2Vec2Model.from_pretrained("fav-kky/wav2vec2-base-cs-80k-ClTRUS")

speech_array, sampling_rate = torchaudio.load("/path/to/audio/file.wav")
inputs = feature_extractor(
    speech_array, 
    sampling_rate=16_000, 
    return_tensors="pt"
)["input_values"][0]

output = model(inputs)
embeddings = output.last_hidden_state.detach().numpy()[0]
```

## Speech recognition results
After fine-tuning, the model scored the following results on public datasets:
- Czech portion of CommonVoice v7.0: **WER = 3.8%**
- Czech portion of VoxPopuli: **WER = 8.8%**

See our paper for details.

## Paper 
The preprint of our paper (accepted to INTERSPEECH 2022) is available at http://arxiv.org/abs/2206.07627

## Citation
If you find this model useful, please cite our paper:
```
@inproceedings{wav2vec2-base-cs-80k-ClTRUS,
  title = {{Exploring Capabilities of Monolingual Audio Transformers using Large Datasets in Automatic Speech Recognition of Czech}},
  author = {
    Jan Lehe\v{c}ka and 
    Jan \v{S}vec and 
    Ale\v{s} Pra\v{z}\'ak and 
    Josef V. Psutka
  },
  booktitle={Proc. Interspeech 2022},
  pages={1831--1835},
  year = {2022},
  doi={10.21437/Interspeech.2022-10439}
}
```

## Related works
- [Transformer-based Automatic Speech Recognition of Formal and Colloquial Czech in MALACH Project](https://arxiv.org/abs/2206.07666)
- [Yehor/wav2vec2-xls-r-base-uk-with-small-lm](https://huggingface.co/Yehor/wav2vec2-xls-r-base-uk-with-small-lm)

{"language": "cs", "license": "cc-by-nc-sa-4.0", "tags": ["Czech", "KKY", "FAV"]}
transformers pytorch wav2vec2 pretraining Czech KKY FAV cs arxiv:2206.07627 arxiv:2206.07666 license:cc-by-nc-sa-4.0 endpoints_compatible region:us
TAGS
#transformers #pytorch #wav2vec2 #pretraining #Czech #KKY #FAV #cs #arxiv-2206.07627 #arxiv-2206.07666 #license-cc-by-nc-sa-4.0 #endpoints_compatible #region-us 


# wav2vec2-base-cs-80k-ClTRUS
Czech language TRransformer from Unlabeled Speech (ClTRUS) is a monolingual Czech Wav2Vec 2.0 base model pre-trained from 80 thousand hours of Czech speech.

This model does not have a tokenizer as it was pretrained on audio alone. In order to use this model for speech recognition, a tokenizer should be created and the model should be fine-tuned on labeled text data. 

Note: This is a checkpoint of the model after 4 epochs over the whole dataset. If you want some earlier or later checkpoints, please feel free to contact the author (jlehecka(at)URL).

## Pretraining data 
More than 80 thousand hours of unlabeled Czech speech:
- recordings from radio (22k hours),
- unlabeled data from VoxPopuli dataset (18.7k hours),
- TV shows (15k hours), 
- shadow speakers (12k hours), 
- sports (5k hours), 
- telephone data (2k hours),
- and a smaller amount of data from several other domains including the public CommonVoice dataset.

## Usage
Inputs must be 16kHz mono audio files.

This model can be used e.g. to extract per-frame contextual embeddings from audio:


## Speech recognition results
After fine-tuning, the model scored the following results on public datasets:
- Czech portion of CommonVoice v7.0: WER = 3.8%
- Czech portion of VoxPopuli: WER = 8.8%

See our paper for details.

## Paper 
The preprint of our paper (accepted to INTERSPEECH 2022) is available at URL

If you find this model useful, please cite our paper:


## Related works
- Transformer-based Automatic Speech Recognition of Formal and Colloquial Czech in MALACH Project
- Yehor/wav2vec2-xls-r-base-uk-with-small-lm

# wav2vec2-base-cs-80k-ClTRUS
Czech language TRransformer from Unlabeled Speech (ClTRUS) is a monolingual Czech Wav2Vec 2.0 base model pre-trained from 80 thousand hours of Czech speech.

This model does not have a tokenizer as it was pretrained on audio alone. In order to use this model for speech recognition, a tokenizer should be created and the model should be fine-tuned on labeled text data. 

Note: This is a checkpoint of the model after 4 epochs over the whole dataset. If you want some earlier or later checkpoints, please feel free to contact the author (jlehecka(at)URL). ## Pretraining data 
More than 80 thousand hours of unlabeled Czech speech:
- recordings from radio (22k hours),
- unlabeled data from VoxPopuli dataset (18.7k hours),
- TV shows (15k hours), 
- shadow speakers (12k hours), 
- sports (5k hours), 
- telephone data (2k hours),
- and a smaller amount of data from several other domains including the public CommonVoice dataset. ## Usage
Inputs must be 16kHz mono audio files.

This model can be used e.g. to extract per-frame contextual embeddings from audio: ## Speech recognition results
After fine-tuning, the model scored the following results on public datasets:
- Czech portion of CommonVoice v7.0: WER = 3.8%
- Czech portion of VoxPopuli: WER = 8.8%

See our paper for details. ## Paper 
The preprint of our paper (accepted to INTERSPEECH 2022) is available at URL

If you find this model useful, please cite our paper: ## Related works
- Transformer-based Automatic Speech Recognition of Formal and Colloquial Czech in MALACH Project
- Yehor/wav2vec2-xls-r-base-uk-with-small-lm
TAGS
#transformers #pytorch #wav2vec2 #pretraining #Czech #KKY #FAV #cs #arxiv-2206.07627 #arxiv-2206.07666 #license-cc-by-nc-sa-4.0 #endpoints_compatible #region-us 
 # wav2vec2-base-cs-80k-ClTRUS
Czech language TRransformer from Unlabeled Speech (ClTRUS) is a monolingual Czech Wav2Vec 2.0 base model pre-trained from 80 thousand hours of Czech speech.

This model does not have a tokenizer as it was pretrained on audio alone. In order to use this model for speech recognition, a tokenizer should be created and the model should be fine-tuned on labeled text data. 

Note: This is a checkpoint of the model after 4 epochs over the whole dataset. If you want some earlier or later checkpoints, please feel free to contact the author (jlehecka(at)URL). ## Pretraining data 
More than 80 thousand hours of unlabeled Czech speech:
- recordings from radio (22k hours),
- unlabeled data from VoxPopuli dataset (18.7k hours),
- TV shows (15k hours), 
- shadow speakers (12k hours), 
- sports (5k hours), 
- telephone data (2k hours),
- and a smaller amount of data from several other domains including the public CommonVoice dataset. ## Usage
Inputs must be 16kHz mono audio files.

This model can be used e.g. to extract per-frame contextual embeddings from audio: ## Speech recognition results
After fine-tuning, the model scored the following results on public datasets:
- Czech portion of CommonVoice v7.0: WER = 3.8%
- Czech portion of VoxPopuli: WER = 8.8%

See our paper for details. ## Paper 
The preprint of our paper (accepted to INTERSPEECH 2022) is available at URL

If you find this model useful, please cite our paper: ## Related works
- Transformer-based Automatic Speech Recognition of Formal and Colloquial Czech in MALACH Project
- Yehor/wav2vec2-xls-r-base-uk-with-small-lm
passage: TAGS
#transformers #pytorch #wav2vec2 #pretraining #Czech #KKY #FAV #cs #arxiv-2206.07627 #arxiv-2206.07666 #license-cc-by-nc-sa-4.0 #endpoints_compatible #region-us 
# wav2vec2-base-cs-80k-ClTRUS
Czech language TRransformer from Unlabeled Speech (ClTRUS) is a monolingual Czech Wav2Vec 2.0 base model pre-trained from 80 thousand hours of Czech speech.

This model does not have a tokenizer as it was pretrained on audio alone. In order to use this model for speech recognition, a tokenizer should be created and the model should be fine-tuned on labeled text data. 

Note: This is a checkpoint of the model after 4 epochs over the whole dataset. If you want some earlier or later checkpoints, please feel free to contact the author (jlehecka(at)URL).## Pretraining data 
More than 80 thousand hours of unlabeled Czech speech:
- recordings from radio (22k hours),
- unlabeled data from VoxPopuli dataset (18.7k hours),
- TV shows (15k hours), 
- shadow speakers (12k hours), 
- sports (5k hours), 
- telephone data (2k hours),
- and a smaller amount of data from several other domains including the public CommonVoice dataset.## Usage
Inputs must be 16kHz mono audio files.

This model can be used e.g. to extract per-frame contextual embeddings from audio:## Speech recognition results
After fine-tuning, the model scored the following results on public datasets:
- Czech portion of CommonVoice v7.0: WER = 3.8%
- Czech portion of VoxPopuli: WER = 8.8%

See our paper for details.## Paper 
The preprint of our paper (accepted to INTERSPEECH 2022) is available at URL

If you find this model useful, please cite our paper:## Related works
- Transformer-based Automatic Speech Recognition of Formal and Colloquial Czech in MALACH Project
- Yehor/wav2vec2-xls-r-base-uk-with-small-lm