Hugging Face's logo

knowledgator
/
gliner-llama-multitask-1B-v1.0

Token Classification
GLiNER
PyTorch
English
NER
information extraction
relation extraction
summarization
sentiment extraction
question-answering
Model card Files Community
1
Ihor commited on
Commit
56baf59
·
verified ·
1 Parent(s): da85c44

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +50 -42
README.md CHANGED
@@ -81,6 +81,7 @@ model = GLiNER.from_pretrained("knowledgator/gliner-llama-1B-v1.0",
81
 
82
  ### Performance:
83
 
 
84
  | Model | Dataset | Precision | Recall | F1 Score | F1 Score (Decimal) |
85
  |------------------------------------|--------------------|-----------|--------|----------|--------------------|
86
  | knowledgator/gliner-multitask-v0.5 | CrossNER_AI | 51.00% | 51.11% | 51.05% | 0.5105 |
@@ -91,14 +92,25 @@ model = GLiNER.from_pretrained("knowledgator/gliner-llama-1B-v1.0",
91
  | | mit-movie | 61.29% | 52.59% | 56.60% | 0.5660 |
92
  | | mit-restaurant | 50.65% | 38.13% | 43.51% | 0.4351 |
93
  | | **Average** | | | | **0.6276** |
94
- | knowledgator/gliner-multitask-v1.0 | CrossNER_AI | 51.00% | 51.11% | 51.05% | 0.5105 |
95
- | | CrossNER_literature | 72.65% | 65.62% | 68.96% | 0.6896 |
96
- | | CrossNER_music | 74.91% | 73.70% | 74.30% | 0.7430 |
97
- | | CrossNER_politics | 78.84% | 77.71% | 78.27% | 0.7827 |
98
- | | CrossNER_science | 69.20% | 65.48% | 67.29% | 0.6729 |
99
- | | mit-movie | 61.29% | 52.59% | 56.60% | 0.5660 |
100
- | | mit-restaurant | 50.65% | 38.13% | 43.51% | 0.4351 |
101
- | | **Average** | | | | **0.6276** |
 
 
 
 
 
 
 
 
 
 
 
102
 
103
  ---
104
  **How to use for relation extraction:**
@@ -331,40 +343,36 @@ for label in classes:
331
 
332
  Our multitask model demonstrates comparable performance on different zero-shot benchmarks to dedicated models to NER task (all labels were lowecased in this testing):
333
 
334
- | Model | Dataset | Precision | Recall | F1 Score | F1 Score (Decimal) |
335
- |------------------------------------|--------------------|-----------|--------|----------|--------------------|
336
- | numind/NuNER_Zero-span | CrossNER_AI | 63.82% | 56.82% | 60.12% | 0.6012 |
337
- | | CrossNER_literature| 73.53% | 58.06% | 64.89% | 0.6489 |
338
- | | CrossNER_music | 72.69% | 67.40% | 69.95% | 0.6995 |
339
- | | CrossNER_politics | 77.28% | 68.69% | 72.73% | 0.7273 |
340
- | | CrossNER_science | 70.08% | 63.12% | 66.42% | 0.6642 |
341
- | | mit-movie | 63.00% | 48.88% | 55.05% | 0.5505 |
342
- | | mit-restaurant | 54.81% | 37.62% | 44.62% | 0.4462 |
343
- | | **Average** | | | | **0.6196** |
344
- | knowledgator/gliner-multitask-v0.5 | CrossNER_AI | 51.00% | 51.11% | 51.05% | 0.5105 |
345
- | | CrossNER_literature | 72.65% | 65.62% | 68.96% | 0.6896 |
346
- | | CrossNER_music | 74.91% | 73.70% | 74.30% | 0.7430 |
347
- | | CrossNER_politics | 78.84% | 77.71% | 78.27% | 0.7827 |
348
- | | CrossNER_science | 69.20% | 65.48% | 67.29% | 0.6729 |
349
- | | mit-movie | 61.29% | 52.59% | 56.60% | 0.5660 |
350
- | | mit-restaurant | 50.65% | 38.13% | 43.51% | 0.4351 |
351
- | | **Average** | | | | **0.6276** |
352
- | urchade/gliner_large-v2.1 | CrossNER_AI | 54.98% | 52.00% | 53.45% | 0.5345 |
353
- | | CrossNER_literature| 59.33% | 56.47% | 57.87% | 0.5787 |
354
- | | CrossNER_music | 67.39% | 66.77% | 67.08% | 0.6708 |
355
- | | CrossNER_politics | 66.07% | 63.76% | 64.90% | 0.6490 |
356
- | | CrossNER_science | 61.45% | 62.56% | 62.00% | 0.6200 |
357
- | | mit-movie | 55.94% | 47.36% | 51.29% | 0.5129 |
358
- | | mit-restaurant | 53.34% | 40.83% | 46.25% | 0.4625 |
359
- | | **Average** | | | | **0.5754** |
360
- | EmergentMethods/gliner_large_news-v2.1| CrossNER_AI | 59.60% | 54.55% | 56.96% | 0.5696 |
361
- | | CrossNER_literature| 65.41% | 56.16% | 60.44% | 0.6044 |
362
- | | CrossNER_music | 67.47% | 63.08% | 65.20% | 0.6520 |
363
- | | CrossNER_politics | 66.05% | 60.07% | 62.92% | 0.6292 |
364
- | | CrossNER_science | 68.44% | 63.57% | 65.92% | 0.6592 |
365
- | | mit-movie | 65.85% | 49.59% | 56.57% | 0.5657 |
366
- | | mit-restaurant | 54.71% | 35.94% | 43.38% | 0.4338 |
367
- | | **Average** | | | | **0.5876** |
368
 
369
 
370
  ### Join Our Discord
 
81
 
82
  ### Performance:
83
 
84
+
85
  | Model | Dataset | Precision | Recall | F1 Score | F1 Score (Decimal) |
86
  |------------------------------------|--------------------|-----------|--------|----------|--------------------|
87
  | knowledgator/gliner-multitask-v0.5 | CrossNER_AI | 51.00% | 51.11% | 51.05% | 0.5105 |
 
92
  | | mit-movie | 61.29% | 52.59% | 56.60% | 0.5660 |
93
  | | mit-restaurant | 50.65% | 38.13% | 43.51% | 0.4351 |
94
  | | **Average** | | | | **0.6276** |
95
+
96
+ | knowledgator/gliner-multitask-v1.0 | CrossNER_AI | 67.15% | 56.10% | 61.13% | 0.6113 |
97
+ | | CrossNER_literature | 71.60% | 64.74% | 68.00% | 0.6800 |
98
+ | | CrossNER_music | 73.57% | 69.29% | 71.36% | 0.7136 |
99
+ | | CrossNER_politics | 77.54% | 76.52% | 77.03% | 0.7703 |
100
+ | | CrossNER_science | 74.54% | 66.00% | 70.01% | 0.7001 |
101
+ | | mit-movie | 61.86% | 42.02% | 50.04% | 0.5004 |
102
+ | | mit-restaurant | 58.87% | 36.67% | 45.19% | 0.4519 |
103
+ | | **Average** | | | | **0.6325** |
104
+
105
+ | knowledgator/gliner-llama-multitask-1B-v1.0 | CrossNER_AI | 63.24% | 55.60% | 59.17% | 0.5917 |
106
+ | | CrossNER_literature | 69.74% | 60.10% | 64.56% | 0.6456 |
107
+ | | CrossNER_music | 74.03% | 67.22% | 70.46% | 0.7046 |
108
+ | | CrossNER_politics | 76.96% | 71.64% | 74.20% | 0.7420 |
109
+ | | CrossNER_science | 73.79% | 63.73% | 68.39% | 0.6839 |
110
+ | | mit-movie | 56.89% | 46.70% | 51.30% | 0.5130 |
111
+ | | mit-restaurant | 48.45% | 38.13% | 42.67% | 0.4267 |
112
+ | | **Average** | | | | **0.6153** |
113
+
114
 
115
  ---
116
  **How to use for relation extraction:**
 
343
 
344
  Our multitask model demonstrates comparable performance on different zero-shot benchmarks to dedicated models to NER task (all labels were lowecased in this testing):
345
 
346
+ Here is the updated table based on the new data:
347
+
348
+ | Dataset | Precision | Recall | F1 Score | F1 Score (Decimal) |
349
+ |------------------------|-----------|--------|----------|--------------------|
350
+ | ACE 2004 | 40.45% | 18.49% | 25.38% | 0.2538 |
351
+ | ACE 2005 | 37.93% | 16.81% | 23.30% | 0.2330 |
352
+ | AnatEM | 41.08% | 29.71% | 34.48% | 0.3448 |
353
+ | Broad Tweet Corpus | 72.68% | 66.58% | 69.50% | 0.6950 |
354
+ | CoNLL 2003 | 70.34% | 68.77% | 69.54% | 0.6954 |
355
+ | CrossNER_AI | 63.24% | 55.60% | 59.17% | 0.5917 |
356
+ | CrossNER_literature | 69.74% | 60.10% | 64.56% | 0.6456 |
357
+ | CrossNER_music | 74.03% | 67.22% | 70.46% | 0.7046 |
358
+ | CrossNER_politics | 76.96% | 71.64% | 74.20% | 0.7420 |
359
+ | CrossNER_science | 73.79% | 63.73% | 68.39% | 0.6839 |
360
+ | FabNER | 35.11% | 16.55% | 22.49% | 0.2249 |
361
+ | FindVehicle | 46.76% | 27.30% | 34.47% | 0.3447 |
362
+ | GENIA_NER | 59.48% | 44.91% | 51.18% | 0.5118 |
363
+ | HarveyNER | 16.52% | 30.12% | 21.34% | 0.2134 |
364
+ | MultiNERD | 54.77% | 86.93% | 67.20% | 0.6720 |
365
+ | Ontonotes | 25.52% | 34.18% | 29.22% | 0.2922 |
366
+ | PolyglotNER | 35.54% | 65.73% | 46.13% | 0.4613 |
367
+ | TweetNER7 | 54.17% | 35.80% | 43.11% | 0.4311 |
368
+ | WikiANN en | 54.97% | 56.83% | 55.88% | 0.5588 |
369
+ | WikiNeural | 71.80% | 85.37% | 78.00% | 0.7800 |
370
+ | bc2gm | 51.17% | 48.71% | 49.91% | 0.4991 |
371
+ | bc4chemd | 50.76% | 68.69% | 58.38% | 0.5838 |
372
+ | bc5cdr | 75.05% | 67.16% | 70.89% | 0.7089 |
373
+ | mit-movie | 56.89% | 46.70% | 51.30% | 0.5130 |
374
+ | mit-restaurant | 48.45% | 38.13% | 42.67% | 0.4267 |
375
+ | ncbi | 66.27% | 57.47% | 61.56% | 0.6156 |
 
 
 
 
376
 
377
 
378
  ### Join Our Discord