
# A-PROOF ICF-domains Classification

## Description

A fine-tuned multi-label classification model that detects 9 [WHO-ICF](https://www.who.int/standards/classifications/international-classification-of-functioning-disability-and-health) domains in clinical text in Dutch. The model is based on a pre-trained Dutch medical language model ([link to be added]()), a RoBERTa model, trained from scratch on clinical notes of the Amsterdam UMC.

## ICF domains
The model can detect 9 domains, which were chosen due to their relevance to recovery from COVID-19:

ICF code | Domain | name in repo
---|---|---
b440 | Respiration functions | ADM
b140 | Attention functions | ATT
d840-d859 | Work and employment | BER
b1300 | Energy level | ENR
d550 | Eating | ETN
d450 | Walking | FAC
b455 | Exercise tolerance functions | INS
b530 | Weight maintenance functions | MBW
b152 | Emotional functions | STM

## Intended uses and limitations
- The model was fine-tuned (trained, validated and tested) on medical records from the Amsterdam UMC (the two academic medical centers of Amsterdam). It might perform differently on text from a different hospital or text from non-hospital sources (e.g. GP records).
- The model was fine-tuned with the [Simple Transformers](https://simpletransformers.ai/) library. This library is based on Transformers but the model cannot be used directly with Transformers `pipeline` and classes; doing so would generate incorrect outputs. For this reason, the API on this page is disabled.

## How to use
To generate predictions with the model, use the [Simple Transformers](https://simpletransformers.ai/) library:
```
from simpletransformers.classification import MultiLabelClassificationModel

model = MultiLabelClassificationModel(
    'roberta',
    'CLTL/icf-domains',
    use_cuda=False,
)

example = 'Nu sinds 5-6 dagen progressieve benauwdheidsklachten (bij korte stukken lopen al kortademig), terwijl dit eerder niet zo was.'
predictions, raw_outputs = model.predict([example])
```
The predictions look like this:
```
[[1, 0, 0, 0, 0, 1, 1, 0, 0]]
```
The indices of the multi-label stand for:
```
[ADM, ATT, BER, ENR, ETN, FAC, INS, MBW, STM]
```
In other words, the above prediction corresponds to assigning the labels ADM, FAC and INS to the example sentence.

The raw outputs look like this:
```
[[0.51907885 0.00268032 0.0030862  0.03066113 0.00616694 0.64720929
  0.67348498 0.0118863  0.0046311 ]]
```
For this model, the threshold at which the prediction for a label flips from 0 to 1 is **0.5**.

## Training data
- The training data consists of clinical notes from medical records (in Dutch) of the Amsterdam UMC. Due to privacy constraints, the data cannot be released.
- The annotation guidelines used for the project can be found [here](https://github.com/cltl/a-proof-zonmw/tree/main/resources/annotation_guidelines).

## Training procedure
The default training parameters of Simple Transformers were used, including:
- Optimizer: AdamW
- Learning rate: 4e-5
- Num train epochs: 1
- Train batch size: 8
- Threshold: 0.5

## Evaluation results
The evaluation is done on a sentence-level (the classification unit) and on a note-level (the aggregated unit which is meaningful for the healthcare professionals).

### Sentence-level
| | ADM | ATT | BER | ENR | ETN | FAC | INS | MBW | STM
|---|---|---|---|---|---|---|---|---|---
precision | 0.98 | 0.98 | 0.56 | 0.96 | 0.92 | 0.84 | 0.89 | 0.79 | 0.70
recall | 0.49 | 0.41 | 0.29 | 0.57 | 0.49 | 0.71 | 0.26 | 0.62 | 0.75
F1-score | 0.66 | 0.58 | 0.35 | 0.72 | 0.63 | 0.76 | 0.41 | 0.70 | 0.72
support | 775 | 39 | 54 | 160 | 382 | 253 | 287 | 125 | 181

### Note-level
| | ADM | ATT | BER | ENR | ETN | FAC | INS | MBW | STM
|---|---|---|---|---|---|---|---|---|---
precision | 1.0 | 1.0 | 0.66 | 0.96 | 0.95 | 0.84 | 0.95 | 0.87 | 0.80
recall | 0.89 | 0.56 | 0.44 | 0.70 | 0.72 | 0.89 | 0.46 | 0.87 | 0.87
F1-score | 0.94 | 0.71 | 0.50 | 0.81 | 0.82 | 0.86 | 0.61 | 0.87 | 0.84
support | 231 | 27 | 34 | 92 | 165 | 95 | 116 | 64 | 94

## Authors and references
### Authors
Jenia Kim, Piek Vossen

### References
TBD
{"language": "nl", "license": "mit", "pipeline_tag": "text-classification", "inference": false}
transformers pytorch roberta text-classification nl license:mit region:us
TAGS
#transformers #pytorch #roberta #text-classification #nl #license-mit #region-us 

A-PROOF ICF-domains Classification
==================================


Description
-----------


A fine-tuned multi-label classification model that detects 9 WHO-ICF domains in clinical text in Dutch. The model is based on a pre-trained Dutch medical language model (link to be added), a RoBERTa model, trained from scratch on clinical notes of the Amsterdam UMC.


ICF domains
-----------


The model can detect 9 domains, which were chosen due to their relevance to recovery from COVID-19:


ICF code: b440, Domain: Respiration functions, name in repo: ADM
ICF code: b140, Domain: Attention functions, name in repo: ATT
ICF code: d840-d859, Domain: Work and employment, name in repo: BER
ICF code: b1300, Domain: Energy level, name in repo: ENR
ICF code: d550, Domain: Eating, name in repo: ETN
ICF code: d450, Domain: Walking, name in repo: FAC
ICF code: b455, Domain: Exercise tolerance functions, name in repo: INS
ICF code: b530, Domain: Weight maintenance functions, name in repo: MBW
ICF code: b152, Domain: Emotional functions, name in repo: STM


Intended uses and limitations
-----------------------------


* The model was fine-tuned (trained, validated and tested) on medical records from the Amsterdam UMC (the two academic medical centers of Amsterdam). It might perform differently on text from a different hospital or text from non-hospital sources (e.g. GP records).
* The model was fine-tuned with the Simple Transformers library. This library is based on Transformers but the model cannot be used directly with Transformers 'pipeline' and classes; doing so would generate incorrect outputs. For this reason, the API on this page is disabled.


How to use
----------


To generate predictions with the model, use the Simple Transformers library:


The predictions look like this:


The indices of the multi-label stand for:


In other words, the above prediction corresponds to assigning the labels ADM, FAC and INS to the example sentence.


The raw outputs look like this:


For this model, the threshold at which the prediction for a label flips from 0 to 1 is 0.5.


Training data
-------------


* The training data consists of clinical notes from medical records (in Dutch) of the Amsterdam UMC. Due to privacy constraints, the data cannot be released.
* The annotation guidelines used for the project can be found here.


Training procedure
------------------


The default training parameters of Simple Transformers were used, including:


* Optimizer: AdamW
* Learning rate: 4e-5
* Num train epochs: 1
* Train batch size: 8
* Threshold: 0.5


Evaluation results
------------------


The evaluation is done on a sentence-level (the classification unit) and on a note-level (the aggregated unit which is meaningful for the healthcare professionals).


### Sentence-level



### Note-level



Authors and references
----------------------


### Authors


Jenia Kim, Piek Vossen


### References


TBD



### Sentence-level ### Note-level



Authors and references
---------------------- ### Authors


Jenia Kim, Piek Vossen ### References


TBD
TAGS
#transformers #pytorch #roberta #text-classification #nl #license-mit #region-us 
 ### Sentence-level ### Note-level



Authors and references
---------------------- ### Authors


Jenia Kim, Piek Vossen ### References


TBD
passage: TAGS
#transformers #pytorch #roberta #text-classification #nl #license-mit #region-us 
### Sentence-level### Note-level



Authors and references
----------------------### Authors


Jenia Kim, Piek Vossen### References


TBD