
## About
Here we share a pretrained BERT model that is aware of math tokens. The math tokens are treated specially and tokenized using [pya0](https://github.com/approach0/pya0), which adds very limited new tokens for latex markup (total vocabulary is just 31,061).

This model is trained on 4 x 2 Tesla V100 with a total batch size of 64, using Math StackExchange data with 2.7 million sentence pairs trained for 7 epochs.

### Usage
Download and try it out
```sh
pip install pya0==0.3.2
wget https://vault.cs.uwaterloo.ca/s/gqstFZmWHCLGXe3/download -O ckpt.tar.gz
mkdir -p ckpt
tar xzf ckpt.tar.gz -C ckpt --strip-components=1
python test.py --test_file test.txt
```

### Test file format
Modify the test examples in `test.txt` to play with it.

The test file is tab-separated, the first column is additional positions you want to mask for the right-side sentence (useful for masking tokens in math markups). A zero means no additional mask positions.

### Example output
![](https://i.imgur.com/xpl87KO.png)

### Upload to huggingface
This repo is hosted on [Github](https://github.com/approach0/azbert), and only mirrored at [huggingface](https://huggingface.co/castorini/azbert-base).

To upload to huggingface, use the `upload2hgf.sh` script.
Before runnig this script, be sure to check:
* check points for model and tokenizer are created under `./ckpt` folder
* model contains all the files needed: `config.json` and `pytorch_model.bin`
* tokenizer contains all the files needed: `added_tokens.json`, `special_tokens_map.json`, `tokenizer_config.json`, `vocab.txt` and `tokenizer.json`
* no `tokenizer_file` field in `tokenizer_config.json` (sometimes it is located locally at `~/.cache`)
* `git-lfs` is installed
* having git-remote named `hgf` reference to `https://huggingface.co/castorini/azbert-base`

{"language": "en", "license": "mit", "tags": ["azbert", "pretraining", "fill-mask"], "widget": [{"text": "$f$ $($ $x$ [MASK] $y$ $)$", "example_title": "mathy"}, {"text": "$x$ [MASK] $x$ $equal$ $2$ $x$", "example_title": "mathy"}, {"text": "Proof by [MASK] that $n$ $fact$ $gt$ $3$ $n$ for $n$ $gt$ $6$", "example_title": "mathy"}, {"text": "Proof by induction that $n$ [MASK] $gt$ $3$ $n$ for $n$ $gt$ $6$", "example_title": "mathy"}, {"text": "The goal of life is [MASK].", "example_title": "philosophical"}]}
transformers pytorch tensorboard bert pretraining azbert fill-mask en license:mit endpoints_compatible region:us
TAGS
#transformers #pytorch #tensorboard #bert #pretraining #azbert #fill-mask #en #license-mit #endpoints_compatible #region-us 


## About
Here we share a pretrained BERT model that is aware of math tokens. The math tokens are treated specially and tokenized using pya0, which adds very limited new tokens for latex markup (total vocabulary is just 31,061).

This model is trained on 4 x 2 Tesla V100 with a total batch size of 64, using Math StackExchange data with 2.7 million sentence pairs trained for 7 epochs.

### Usage
Download and try it out


### Test file format
Modify the test examples in 'URL' to play with it.

The test file is tab-separated, the first column is additional positions you want to mask for the right-side sentence (useful for masking tokens in math markups). A zero means no additional mask positions.

### Example output
![](https://i.URL

### Upload to huggingface
This repo is hosted on Github, and only mirrored at huggingface.

To upload to huggingface, use the 'URL' script.
Before runnig this script, be sure to check:
* check points for model and tokenizer are created under './ckpt' folder
* model contains all the files needed: 'URL' and 'pytorch_model.bin'
* tokenizer contains all the files needed: 'added_tokens.json', 'special_tokens_map.json', 'tokenizer_config.json', 'URL' and 'URL'
* no 'tokenizer_file' field in 'tokenizer_config.json' (sometimes it is located locally at '~/.cache')
* 'git-lfs' is installed
* having git-remote named 'hgf' reference to 'URL

## About
Here we share a pretrained BERT model that is aware of math tokens. The math tokens are treated specially and tokenized using pya0, which adds very limited new tokens for latex markup (total vocabulary is just 31,061).

This model is trained on 4 x 2 Tesla V100 with a total batch size of 64, using Math StackExchange data with 2.7 million sentence pairs trained for 7 epochs. ### Usage
Download and try it out ### Test file format
Modify the test examples in 'URL' to play with it.

The test file is tab-separated, the first column is additional positions you want to mask for the right-side sentence (useful for masking tokens in math markups). A zero means no additional mask positions. ### Example output
![](https://i.URL ### Upload to huggingface
This repo is hosted on Github, and only mirrored at huggingface.

To upload to huggingface, use the 'URL' script.
Before runnig this script, be sure to check:
* check points for model and tokenizer are created under './ckpt' folder
* model contains all the files needed: 'URL' and 'pytorch_model.bin'
* tokenizer contains all the files needed: 'added_tokens.json', 'special_tokens_map.json', 'tokenizer_config.json', 'URL' and 'URL'
* no 'tokenizer_file' field in 'tokenizer_config.json' (sometimes it is located locally at '~/.cache')
* 'git-lfs' is installed
* having git-remote named 'hgf' reference to 'URL
TAGS
#transformers #pytorch #tensorboard #bert #pretraining #azbert #fill-mask #en #license-mit #endpoints_compatible #region-us 
 ## About
Here we share a pretrained BERT model that is aware of math tokens. The math tokens are treated specially and tokenized using pya0, which adds very limited new tokens for latex markup (total vocabulary is just 31,061).

This model is trained on 4 x 2 Tesla V100 with a total batch size of 64, using Math StackExchange data with 2.7 million sentence pairs trained for 7 epochs. ### Usage
Download and try it out ### Test file format
Modify the test examples in 'URL' to play with it.

The test file is tab-separated, the first column is additional positions you want to mask for the right-side sentence (useful for masking tokens in math markups). A zero means no additional mask positions. ### Example output
![](https://i.URL ### Upload to huggingface
This repo is hosted on Github, and only mirrored at huggingface.

To upload to huggingface, use the 'URL' script.
Before runnig this script, be sure to check:
* check points for model and tokenizer are created under './ckpt' folder
* model contains all the files needed: 'URL' and 'pytorch_model.bin'
* tokenizer contains all the files needed: 'added_tokens.json', 'special_tokens_map.json', 'tokenizer_config.json', 'URL' and 'URL'
* no 'tokenizer_file' field in 'tokenizer_config.json' (sometimes it is located locally at '~/.cache')
* 'git-lfs' is installed
* having git-remote named 'hgf' reference to 'URL
passage: TAGS
#transformers #pytorch #tensorboard #bert #pretraining #azbert #fill-mask #en #license-mit #endpoints_compatible #region-us 
## About
Here we share a pretrained BERT model that is aware of math tokens. The math tokens are treated specially and tokenized using pya0, which adds very limited new tokens for latex markup (total vocabulary is just 31,061).

This model is trained on 4 x 2 Tesla V100 with a total batch size of 64, using Math StackExchange data with 2.7 million sentence pairs trained for 7 epochs.### Usage
Download and try it out### Test file format
Modify the test examples in 'URL' to play with it.

The test file is tab-separated, the first column is additional positions you want to mask for the right-side sentence (useful for masking tokens in math markups). A zero means no additional mask positions.### Example output
![](https://i.URL### Upload to huggingface
This repo is hosted on Github, and only mirrored at huggingface.

To upload to huggingface, use the 'URL' script.
Before runnig this script, be sure to check:
* check points for model and tokenizer are created under './ckpt' folder
* model contains all the files needed: 'URL' and 'pytorch_model.bin'
* tokenizer contains all the files needed: 'added_tokens.json', 'special_tokens_map.json', 'tokenizer_config.json', 'URL' and 'URL'
* no 'tokenizer_file' field in 'tokenizer_config.json' (sometimes it is located locally at '~/.cache')
* 'git-lfs' is installed
* having git-remote named 'hgf' reference to 'URL