doberst commited on
Commit
4016947
·
verified ·
1 Parent(s): 9b7dd72

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +17 -1
README.md CHANGED
@@ -12,7 +12,23 @@ To pull the model via API:
12
 
13
  from huggingface_hub import snapshot_download
14
  snapshot_download("llmware/bling-qwen-nano-tool", local_dir="/path/on/your/machine/", local_dir_use_symlinks=False)
15
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
16
 
17
  Load in your favorite GGUF inference engine, or try with llmware as follows:
18
 
 
12
 
13
  from huggingface_hub import snapshot_download
14
  snapshot_download("llmware/bling-qwen-nano-tool", local_dir="/path/on/your/machine/", local_dir_use_symlinks=False)
15
+
16
+
17
+ ## Benchmark Tests
18
+
19
+ Evaluated against the benchmark test: RAG-Instruct-Benchmark-Tester
20
+ 1 Test Run with sample=False & temperature=0.0 (deterministic output) - 1 point for correct answer, 0.5 point for partial correct or blank / NF, 0.0 points for incorrect, and -1 points for hallucinations.
21
+
22
+ --Accuracy Score: **81.0** correct out of 100
23
+ --Not Found Classification: 65.0%
24
+ --Boolean: 62.5%
25
+ --Math/Logic: 42.5%
26
+ --Complex Questions (1-5): 3 (Average for ~1B model)
27
+ --Summarization Quality (1-5): 3 (Average)
28
+ --Hallucinations: No hallucinations observed in test runs.
29
+
30
+ For test run results (and good indicator of target use cases), please see the files ("core_rag_test" and "answer_sheet" in this repo).
31
+
32
 
33
  Load in your favorite GGUF inference engine, or try with llmware as follows:
34