Add model

README.md

@@ -0,0 +1,78 @@
+---
+
+language: ja
+
+license: cc-by-sa-4.0
+
+tags:
+
+- finance
+
+datasets:
+
+- securities reports
+- summaries of financial results
+
+widget:
+
+- text: 流動[MASK]は、1億円となりました。
+
+---
+
+# Additional pretrained BERT base Japanese finance 
+
+This is a [BERT](https://github.com/google-research/bert) model pretrained on texts in the Japanese language.
+
+The codes for the pretraining are available at [retarfi/language-pretraining](https://github.com/retarfi/language-pretraining/tree/v1.0).
+
+## Model architecture
+
+The model architecture is the same as BERT small in the [original BERT paper](https://arxiv.org/abs/1810.04805); 12 layers, 768 dimensions of hidden states, and 12 attention heads.
+
+## Training Data
+
+The models are additionally trained on financial corpus from [Tohoku University's BERT base Japanese model (cl-tohoku/bert-base-japanese)](https://huggingface.co/cl-tohoku/bert-base-japanese).
+
+The financial corpus consists of 2 corpora:
+
+- Summaries of financial results from October 9, 2012, to December 31, 2020
+- Securities reports from February 8, 2018, to December 31, 2020
+
+The financial corpus file consists of approximately 27M sentences.
+
+
+## Tokenization
+
+You can use tokenizer [Tohoku University's BERT base Japanese model (cl-tohoku/bert-base-japanese)](https://huggingface.co/cl-tohoku/bert-base-japanese).
+
+You can use the tokenizer:
+
+```
+tokenizer = transformers.BertJapaneseTokenizer.from_pretrained('cl-tohoku/bert-base-japanese')
+```
+
+## Training
+
+The models are trained with the same configuration as BERT base in the [original BERT paper](https://arxiv.org/abs/1810.04805); 512 tokens per instance, 256 instances per batch, and 1M training steps.
+
+## Citation
+
+**There will be another paper for this pretrained model. Be sure to check here again when you cite.**
+
+```
+@inproceedings{bert_electra_japanese,
+  title = {Construction and Validation of a Pre-Training and Additional Pre-Training Financial Language Model}
+  author = {Masahiro Suzuki and Hiroki Sakaji and Masanori Hirano and Kiyoshi Izumi},
+  month = {mar},
+  year = {2022},
+  booktitle = {"Proceedings of JSAI Special Interest Group on Financial Infomatics (SIG-FIN) 28"}
+}
+```
+
+## Licenses
+
+The pretrained models are distributed under the terms of the [Creative Commons Attribution-ShareAlike 4.0](https://creativecommons.org/licenses/by-sa/4.0/).
+
+## Acknowledgments
+
+This work was supported by JSPS KAKENHI Grant Number JP21K12010 and JST-Mirai Program Grant Number JPMJMI20B1.

config.json

@@ -0,0 +1,24 @@
+{
+  "architectures": [
+    "BertForPreTraining"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "tokenizer_class": "BertJapaneseTokenizer",
+  "transformers_version": "4.7.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 32000
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef069c1fa9805436139e56236f411eb94bd6d73b63ffdb75702962baff942222
+size 445063670