Update README.md

README.md

@@ -28,28 +28,28 @@ model-index:
       revision: 172e61bb1dd20e43903f4c51e5cbec61ec9ae6e6
     metrics:
     - type: accuracy
-      value: 0.7336065573770492
+      value: 0.73
       name: Accuracy 'Bezeichnung'
     - type: precision
-      value: 0.6611111111111111
+      value: 0.66
       name: Precision 'Bezeichnung' (macro)
     - type: recall
-      value: 0.7056652046783626
+      value: 0.71
       name: Recall 'Bezeichnung' (macro)
     - type: f1
-      value: 0.6674970889256604
+      value: 0.68
       name: F1 'Bezeichnung' (macro)
     - type: accuracy
-      value: 0.8934426229508197
+      value: 0.89
       name: Accuracy 'Thema'
     - type: precision
-      value: 0.902382942746851
+      value: 0.90
       name: Precision 'Thema' (macro)
     - type: recall
-      value: 0.8909340386389567
+      value: 0.89
       name: Recall 'Thema' (macro)
     - type: f1
-      value: 0.8768881364249785
+      value: 0.88
       name: F1 'Thema' (macro)
 ---
 
@@ -61,6 +61,19 @@ model-index:
 
 This model is based on [bert-base-german-cased](https://huggingface.co/bert-base-cased) and fine-tuned on [and-effect/mdk_gov_data_titles_clf](https://huggingface.co/datasets/and-effect/mdk_gov_data_titles_clf).
 
+It was created as part of the Bertelsmann Foundation's Musterdatenkatalog (MDK) project (See their website [here](https://www.bertelsmann-stiftung.de/de/unsere-projekte/smart-country/musterdatenkatalog)).
+The main intent of the MDK project was to classify open data into a taxonomy to help give an overview of already published data.
+It can help municipalities in Germany, as well as data analysts and journalists, to see which cities have already published data sets and what might be missing.
+The project uses a taxonomy to classify the data and the model was specifically trained for the project and the classification task. It thus has a clear intended downstream task and should be used with the mentioned taxonomy.
+
+**Information about the underlying taxonomy:**
+The used taxonomy 'Musterdatenkatalog' has two levels: 'Thema' and 'Bezeichnung' which roughly translates to topic and label. There are 25 entries for the top level ranging from topics such as 'Finanzen' (finance) to 'Gesundheit' (health). 
+The second level, 'Bezeichnung' (label) goes into more detail and would for example contain 'Krankenhaus' (hospital) in the case of the topic being health. The second level contains 241 labels. The combination of topic and label (Thema + Bezeichnung) creates a 'Musterdatensatz'. 
+
+One can classify the data into the topics or the labels, results for both are presented down below. Although matching to other taxonomies is provdided in the published rdf version of the taxonomy (todo), the model is tailored to this taxonomy.
+
+
+
 - **Developed by:** and-effect
 - **Model type:** Text Classification
 - **Language(s) (NLP):** de
@@ -137,14 +150,19 @@ print(sentence_embeddings)
 
 <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
 
-The model is intended to classify open source dataset titles from german municipalities. More information on the Taxonomy (classification categories) and the Project can be found on [More Information Needed].
+The model is intended to classify open source dataset titles from german municipalities. The model is specifically tailored for this task and uses a specific taxonomy.
+More information on the taxonomy (classification categories) and the Project can be found on the [project website](https://www.bertelsmann-stiftung.de/de/unsere-projekte/smart-country/musterdatenkatalog).
+
 
 
 # Bias, Risks, and Limitations
 
 <!-- This section is meant to convey both technical and sociotechnical limitations. -->
 
-The model has some limititations. The model has some limitations in terms of the downstream task. \n 1. **Distribution of classes**: The dataset trained on is small, but at the same time the number of classes is very high. Thus, for some classes there are only a few examples (more information about the class distribution of the training data can be found here). Consequently, the performance for smaller classes may not be as good as for the majority classes. Accordingly, the evaluation is also limited. \n 2. **Systematic problems**: some subjects could not be correctly classified systematically. One example is the embedding of titles containing 'Corona'. In none of the evaluation cases could the titles be embedded in such a way that they corresponded to their true names. Another systematic example is the embedding and classification of titles related to 'migration'. \n 3. **Generalization of the model**: by using semantic search, the model is able to classify titles into new categories that have not been trained, but the model is not tuned for this and therefore the performance of the model for unseen classes is likely to be limited.
+The model has some limititations. The model has some limitations in terms of the downstream task. 
+1. **Distribution of classes**: The dataset trained on is small, but at the same time the number of classes is very high. Thus, for some classes there are only a few examples (more information about the class distribution of the training data can be found here). Consequently, the performance for smaller classes may not be as good as for the majority classes. Accordingly, the evaluation is also limited.
+2. **Systematic problems**: some subjects could not be correctly classified systematically. One example is the embedding of titles containing 'Corona'. In none of the evaluation cases could the titles be embedded in such a way that they corresponded to their true names. Another systematic example is the embedding and classification of titles related to 'migration'.
+3. **Generalization of the model**: by using semantic search, the model is able to classify titles into new categories that have not been trained, but the model is not tuned for this and therefore the performance of the model for unseen classes is likely to be limited.
 
 ## Recommendations
 
@@ -213,17 +231,30 @@ The evaluation data can be found [here](https://huggingface.co/datasets/and-effe
 
 ### Metrics
 
-The model performance is tested with fours metrices. Accuracy, Precision, Recall and F1 Score. A lot of classes were not predicted and are thus set to zero for the calculation of precision, recall and f1 score. For these metrices the additional calucations were performed exluding classes with less than two predictions for the level 'Bezeichnung' (see in table results 'Bezeichnung II'. Although intepretation of these results should be interpreted with caution, because they do not represent all classes.
+The model performance is tested with four metrics: Accuracy, Precision, Recall and F1 Score. Although the data is imbalanced accuracy was still used as the imbalance accurately represents the tendency for more entries for some classes, for example 'Raumplanung - Bebauungsplan'.
+
+A lot of classes were not predicted and are thus set to zero for the calculation of precision, recall and f1 score. 
+For these metrices additional calculations were performed. These are denoted with 'II' in the table and excluded the classes with less than two predictions for the level 'Bezeichnung'. 
+One must be careful when interpreting the results of these calculations though as they do not give any information about the classes left out.
+
+The tasks denoted with 'I' include all classes. 
+
+The tasks are split not only into either including all classes ('I') or not ('II'), they are also divided into a task on 'Bezeichnung' or 'Thema'. 
+As previously mentioned this has to do with the underlying taxonomy. The task on 'Thema' is performed on the first level of the taxonomy with 25 classes, the task on 'Bezeichnung' is performed on the second level which has 241 classes.
+
 
 ## Results
 
 | ***task*** | ***acccuracy*** | ***precision (macro)*** | ***recall (macro)*** | ***f1 (macro)*** |
 |-----|-----|-----|-----|-----|
-| Test dataset 'Bezeichnung' I | 0.7336065573770492 | 0.6611111111111111 | 0.7056652046783626 | 0.6674970889256604 |
-| Test dataset 'Thema' I | 0.8934426229508197 | 0.902382942746851 | 0.8909340386389567 | 0.8768881364249785 |
-| Test dataset 'Bezeichnung' II | 0.7336065573770492 | 0.5829457364341085 | 0.8229090167278661 | 0.6544072311514172 |
-| Validation dataset 'Bezeichnung' I | 0.5148514851485149 | 0.346125116713352 | 0.3553921568627451 | 0.33252525252525256 |
-| Validation dataset 'Thema' I | 0.7722772277227723 | 0.5908392682586232 | 0.6784524126899494 | 0.5962308463774738 |
-| Validation dataset 'Bezeichnung' II | 0.5148514851485149 | 0.5768253968253969 | 0.6916666666666667 | 0.592808080808081 |
+| Test dataset 'Bezeichnung' I | 0.73 (.82)* | 0.66 | 0.71 | 0.67 |
+| Test dataset 'Thema' I | 0.89 (.92)* | 0.90 | 0.89 | 0.88 |
+| Test dataset 'Bezeichnung' II | 0.73 | 0.58 | 0.82 | 0.65 |
+| Validation dataset 'Bezeichnung' I | 0.51 | 0.35 | 0.36 | 0.33 |
+| Validation dataset 'Thema' I | 0.77 | 0.59 | 0.68 | 0.60 |
+| Validation dataset 'Bezeichnung' II | 0.51 | 0.58 | 0.69 | 0.59 |
 
+\* the accuracy in brackets was calculated with a manual analysis. This was done to check for data entries that could for example be part of more than one class and thus were actually correctly classified by the algorithm. 
+In this step the correct labeling of the test data was also checked again for possible mistakes and resulted in a better performance. 
 
+The validation dataset was created manually to check certain classes