Aarushhh commited on
Commit
2e3abef
·
verified ·
1 Parent(s): 4f22b2d

lower final logit softcapping from 30 to 15 ( as kellerjordans x post reccomends)

Browse files
Files changed (1) hide show
  1. config.json +1 -1
config.json CHANGED
@@ -14,7 +14,7 @@
14
  "bos_token_id": 1,
15
  "eos_token_id": 2,
16
  "ep_size": 1,
17
- "final_logit_softcapping": 30.0,
18
  "first_k_dense_replace": 3,
19
  "hidden_act": "relu2",
20
  "hidden_size": 400,
 
14
  "bos_token_id": 1,
15
  "eos_token_id": 2,
16
  "ep_size": 1,
17
+ "final_logit_softcapping": 15.0,
18
  "first_k_dense_replace": 3,
19
  "hidden_act": "relu2",
20
  "hidden_size": 400,