Adamiros commited on
Commit
ae10add
·
verified ·
1 Parent(s): 0098e4c

Update src/display/about.py

Browse files
Files changed (1) hide show
  1. src/display/about.py +21 -10
src/display/about.py CHANGED
@@ -55,16 +55,27 @@ For more information on the included benchmarks and instructions on evaluating y
55
 
56
  # Which evaluations are you running? how can people reproduce what you have?
57
  LLM_BENCHMARKS_TEXT = f"""
58
- ## Basic Information
59
- The goal of this project is to provide a comprehensive and practical benchmark for evaluating Czech language models. This benchmark consists of 15 selected test tasks containing test data in the Czech language. It includes both original Czech datasets and machine translations of popular datasets such as ARC, GSM8K, MMLU, and TruthfulQA. A list of all datasets can be found at [GitHub](https://github.com/jirkoada/czechbench_eval_harness/tree/main/lm_eval/tasks/czechbench#readme)
60
-
61
- Key Features and Benefits:
62
- - **Tailored for the Czech Language:** The benchmark includes both original Czech datasets and adapted versions of international datasets, ensuring relevant evaluation of model performance in the Czech context.
63
- - **Wide Range of Tasks:** It contains 15 different tasks that cover various aspects of language understanding and text generation, enabling a comprehensive assessment of the model's capabilities.
64
- - **Ease of Use:** The benchmark is designed to be easily integrated into your development process, saving time and resources during model testing and improvement.
65
- - **Up-to-date and Relevant:** We regularly update our datasets to reflect the latest findings and trends in language model development.
66
-
67
- By using this benchmark, you will gain deep insights into the strengths and weaknesses of your models, allowing you to better focus on key areas for optimization. This will not only improve the performance of your models but also enhance their real-world deployment in various Czech contexts.
 
 
 
 
 
 
 
 
 
 
 
68
 
69
  ## Evaluation Process
70
 
 
55
 
56
  # Which evaluations are you running? how can people reproduce what you have?
57
  LLM_BENCHMARKS_TEXT = f"""
58
+ ## Included benchmarks
59
+
60
+ All currently supported benchmarks are listed in the table below:
61
+
62
+ | Dataset | Language | Task type | Metrics | Samples | Task ID |
63
+ | ------------------------------------------------------------ | ----------------------------- | -------------------------- | -------------- | ------: | --------------- |
64
+ | [AGREE](https://github.com/jirkoada/czechbench_eval_harness/tree/main/lm_eval/tasks/czechbenchagree_cs) | CS (Original) | Subject-verb agreement | Acc | 627 | agree_cs |
65
+ | [ANLI](https://github.com/jirkoada/czechbench_eval_harness/tree/main/lm_eval/tasks/czechbenchanli_cs) | CS (Translated) | Natural Language Inference | Acc, Macro F1 | 1200 | anli_cs |
66
+ | [ARC Challenge](https://github.com/jirkoada/czechbench_eval_harness/tree/main/lm_eval/tasks/czechbencharc_cs) | CS (Translated) | Knowledge-Based QA | Acc | 1172 | arc_cs |
67
+ | [ARC Easy](https://github.com/jirkoada/czechbench_eval_harness/tree/main/lm_eval/tasks/czechbencharc_cs) | CS (Translated) | Knowledge-Based QA | Acc | 2376 | arc_cs |
68
+ | [Belebele](https://github.com/jirkoada/czechbench_eval_harness/tree/main/lm_eval/tasks/czechbenchbelebele_cs) | CS (Professional translation) | Reading Comprehension / QA | Acc | 895 | belebele_cs |
69
+ | [CTKFacts](https://github.com/jirkoada/czechbench_eval_harness/tree/main/lm_eval/tasks/czechbenchctkfacts_cs) | CS (Original) | Natural Language Inference | Acc, Macro F1 | 558 | ctkfacts_cs |
70
+ | [Czech News](https://github.com/jirkoada/czechbench_eval_harness/tree/main/lm_eval/tasks/czechbenchczechnews_cs) | CS (Original) | News Topic Classification | Acc, Macro F1 | 1000 | czechnews_cs |
71
+ | [Facebook Comments](https://github.com/jirkoada/czechbench_eval_harness/tree/main/lm_eval/tasks/czechbenchfb_comments_cs) | CS (Original) | Sentiment Analysis | Acc, Macro F1 | 1000 | fb_comments_cs |
72
+ | [GSM8K](https://github.com/jirkoada/czechbench_eval_harness/tree/main/lm_eval/tasks/czechbenchgsm8k_cs) | CS (Translated) | Mathematical inference | EM Acc | 1319 | gsm8k_cs |
73
+ | [Klokánek](https://github.com/jirkoada/czechbench_eval_harness/tree/main/lm_eval/tasks/czechbenchklokanek_cs) | CS (Original) | Math/Logical Inference | Acc | 808 | klokanek_cs |
74
+ | [Mall Reviews](https://github.com/jirkoada/czechbench_eval_harness/tree/main/lm_eval/tasks/czechbenchmall_reviews_cs) | CS (Original) | Sentiment Analysis | Acc, Macro F1 | 3000 | mall_reviews_cs |
75
+ | [MMLU](https://github.com/jirkoada/czechbench_eval_harness/tree/main/lm_eval/tasks/czechbenchmmlu_cs) | CS (Translated) | Knowledge-Based QA | Acc | 12408 | mmlu_cs |
76
+ | [SQAD](https://github.com/jirkoada/czechbench_eval_harness/tree/main/lm_eval/tasks/czechbenchsqad_cs) | CS (Original) | Reading Comprehension / QA | EM Acc, BoW F1 | 843 | sqad_cs |
77
+ | [Subjectivity](https://github.com/jirkoada/czechbench_eval_harness/tree/main/lm_eval/tasks/czechbenchsubjectivity_cs) | CS (Original) | Subjectivity Analysis | Acc, Macro F1 | 2000 | subjectivity_cs |
78
+ | [TruthfulQA](https://github.com/jirkoada/czechbench_eval_harness/tree/main/lm_eval/tasks/czechbenchtruthfulqa_cs) | CS (Translated) | Knowledge-Based QA | Acc | 813 | truthfulqa_cs |
79
 
80
  ## Evaluation Process
81