# Background
This model was built on Microsoft's BERT trained on PubMed uncased database (`microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext`). A number of (~500) radiology reports for staging nasopharyngeal carcinoma (NPC) written in our center by board-certified radiologist were retrospectively retrieved with ethics approval . To focus on NPC, incidental findings and unrelated observations are removed prior to training. In addition, the abbreviations for structures were replaced by the original words to facilitate the model of learning suffixes and prefixes that might indicate geographical locations (e.g. L neck -> left neck, IJC -> internal jugular chain). 

A tokenizer was trained based on the original PubMed version, and the radiology reports were used to fine tune the PubMedBert. This fine tuned model has the weakness of unable to identify phrase or multi-word nouns, e.g. "nodal metastatases" is considered two separate words such that the BERT module tends to fill "nodes" when these two words are masked.

This model serve as a pilot analysis of whether it is possible to adopt a transformer based deep learning for radiology report corpus of NPC. 

# Affiliations
Imaging and Interventional Radiology, 

Chinese University of Hong Kong


# Training Losses
| Epoch | Training Loss | Validation Loss |
|-------|---------------|-----------------|
| 1     | No log        | 3.474347        |
| 2     | No log        | 3.174083        |
| 3     | No log        | 2.944307        |
| 4     | No log        | 2.674384        |
| 5     | No log        | 2.574261        |
| 6     | No log        | 2.390012        |
| 7     | No log        | 2.209419        |
| 8     | 2.464700      | 2.107448        |
| 9     | 2.464700      | 1.974744        |
| 10    | 2.464700      | 1.841606        |
| 11    | 2.464700      | 1.783265        |
| 12    | 2.464700      | 1.674914        |
| 13    | 2.464700      | 1.572721        |
| 14    | 2.464700      | 1.546106        |
| 15    | 2.464700      | 1.507173        |
| 16    | 1.153500      | 1.445264        |
| 17    | 1.153500      | 1.394671        |
| 18    | 1.153500      | 1.345976        |
| 19    | 1.153500      | 1.312650        |
| 20    | 1.153500      | 1.256743        |
| 21    | 1.153500      | 1.233211        |
| 22    | 1.153500      | 1.213525        |
| 23    | 1.153500      | 1.182824        |
| 24    | 0.681100      | 1.164411        |
| 25    | 0.681100      | 1.128899        |
| 26    | 0.681100      | 1.145166        |
| 27    | 0.681100      | 1.079617        |
| 28    | 0.681100      | 1.087909        |
| 29    | 0.681100      | 1.102839        |
| 30    | 0.681100      | 1.066386        |
| 31    | 0.681100      | 1.094807        |
| 32    | 0.478400      | 1.060072        |
| 33    | 0.478400      | 1.016879        |
| 34    | 0.478400      | 0.999808        |
| 35    | 0.478400      | 0.987576        |
| 36    | 0.478400      | 1.011713        |
| 37    | 0.478400      | 0.996884        |
| 38    | 0.478400      | 1.018533        |
| 39    | 0.478400      | 1.015250        |
| 40    | 0.378400      | 0.945075        |
| 41    | 0.378400      | 0.950782        |
| 42    | 0.378400      | 1.004242        |
| 43    | 0.378400      | 0.984930        |
| 44    | 0.378400      | 0.966999        |
| 45    | 0.378400      | 0.988593        |
| 46    | 0.378400      | 0.970504        |
| 47    | 0.378400      | 0.976804        |
| 48    | 0.339400      | 1.001518        |
| 49    | 0.339400      | 0.986024        |
| 50    | 0.339400      | 0.987911        |
{"license": "mit", "tags": ["Nasopharyngeal carcinoma", "Cancer"], "thumbnail": "url to a thumbnail used in social sharing", "widget": [{"text": "Nasopharyngeal carcinoma confined in the [MASK].", "example_title": "Example 1"}, {"text": "Nodal metastases in the left side of the [MASK].", "example_title": "Example 2"}, {"text": "Small bilateral cervical [MASK] with unusual distribution.", "example_title": "Example 3"}]}
transformers pytorch bert fill-mask Nasopharyngeal carcinoma Cancer license:mit autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #bert #fill-mask #Nasopharyngeal carcinoma #Cancer #license-mit #autotrain_compatible #endpoints_compatible #region-us 

Background
==========


This model was built on Microsoft's BERT trained on PubMed uncased database ('microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext'). A number of (~500) radiology reports for staging nasopharyngeal carcinoma (NPC) written in our center by board-certified radiologist were retrospectively retrieved with ethics approval . To focus on NPC, incidental findings and unrelated observations are removed prior to training. In addition, the abbreviations for structures were replaced by the original words to facilitate the model of learning suffixes and prefixes that might indicate geographical locations (e.g. L neck -> left neck, IJC -> internal jugular chain).


A tokenizer was trained based on the original PubMed version, and the radiology reports were used to fine tune the PubMedBert. This fine tuned model has the weakness of unable to identify phrase or multi-word nouns, e.g. "nodal metastatases" is considered two separate words such that the BERT module tends to fill "nodes" when these two words are masked.


This model serve as a pilot analysis of whether it is possible to adopt a transformer based deep learning for radiology report corpus of NPC.


Affiliations
============


Imaging and Interventional Radiology,


Chinese University of Hong Kong


Training Losses
===============


Epoch: 1, Training Loss: No log, Validation Loss: 3.474347
Epoch: 2, Training Loss: No log, Validation Loss: 3.174083
Epoch: 3, Training Loss: No log, Validation Loss: 2.944307
Epoch: 4, Training Loss: No log, Validation Loss: 2.674384
Epoch: 5, Training Loss: No log, Validation Loss: 2.574261
Epoch: 6, Training Loss: No log, Validation Loss: 2.390012
Epoch: 7, Training Loss: No log, Validation Loss: 2.209419
Epoch: 8, Training Loss: 2.464700, Validation Loss: 2.107448
Epoch: 9, Training Loss: 2.464700, Validation Loss: 1.974744
Epoch: 10, Training Loss: 2.464700, Validation Loss: 1.841606
Epoch: 11, Training Loss: 2.464700, Validation Loss: 1.783265
Epoch: 12, Training Loss: 2.464700, Validation Loss: 1.674914
Epoch: 13, Training Loss: 2.464700, Validation Loss: 1.572721
Epoch: 14, Training Loss: 2.464700, Validation Loss: 1.546106
Epoch: 15, Training Loss: 2.464700, Validation Loss: 1.507173
Epoch: 16, Training Loss: 1.153500, Validation Loss: 1.445264
Epoch: 17, Training Loss: 1.153500, Validation Loss: 1.394671
Epoch: 18, Training Loss: 1.153500, Validation Loss: 1.345976
Epoch: 19, Training Loss: 1.153500, Validation Loss: 1.312650
Epoch: 20, Training Loss: 1.153500, Validation Loss: 1.256743
Epoch: 21, Training Loss: 1.153500, Validation Loss: 1.233211
Epoch: 22, Training Loss: 1.153500, Validation Loss: 1.213525
Epoch: 23, Training Loss: 1.153500, Validation Loss: 1.182824
Epoch: 24, Training Loss: 0.681100, Validation Loss: 1.164411
Epoch: 25, Training Loss: 0.681100, Validation Loss: 1.128899
Epoch: 26, Training Loss: 0.681100, Validation Loss: 1.145166
Epoch: 27, Training Loss: 0.681100, Validation Loss: 1.079617
Epoch: 28, Training Loss: 0.681100, Validation Loss: 1.087909
Epoch: 29, Training Loss: 0.681100, Validation Loss: 1.102839
Epoch: 30, Training Loss: 0.681100, Validation Loss: 1.066386
Epoch: 31, Training Loss: 0.681100, Validation Loss: 1.094807
Epoch: 32, Training Loss: 0.478400, Validation Loss: 1.060072
Epoch: 33, Training Loss: 0.478400, Validation Loss: 1.016879
Epoch: 34, Training Loss: 0.478400, Validation Loss: 0.999808
Epoch: 35, Training Loss: 0.478400, Validation Loss: 0.987576
Epoch: 36, Training Loss: 0.478400, Validation Loss: 1.011713
Epoch: 37, Training Loss: 0.478400, Validation Loss: 0.996884
Epoch: 38, Training Loss: 0.478400, Validation Loss: 1.018533
Epoch: 39, Training Loss: 0.478400, Validation Loss: 1.015250
Epoch: 40, Training Loss: 0.378400, Validation Loss: 0.945075
Epoch: 41, Training Loss: 0.378400, Validation Loss: 0.950782
Epoch: 42, Training Loss: 0.378400, Validation Loss: 1.004242
Epoch: 43, Training Loss: 0.378400, Validation Loss: 0.984930
Epoch: 44, Training Loss: 0.378400, Validation Loss: 0.966999
Epoch: 45, Training Loss: 0.378400, Validation Loss: 0.988593
Epoch: 46, Training Loss: 0.378400, Validation Loss: 0.970504
Epoch: 47, Training Loss: 0.378400, Validation Loss: 0.976804
Epoch: 48, Training Loss: 0.339400, Validation Loss: 1.001518
Epoch: 49, Training Loss: 0.339400, Validation Loss: 0.986024
Epoch: 50, Training Loss: 0.339400, Validation Loss: 0.987911



TAGS
#transformers #pytorch #bert #fill-mask #Nasopharyngeal carcinoma #Cancer #license-mit #autotrain_compatible #endpoints_compatible #region-us 

passage: TAGS
#transformers #pytorch #bert #fill-mask #Nasopharyngeal carcinoma #Cancer #license-mit #autotrain_compatible #endpoints_compatible #region-us 
