tokenizer note

README.md

@@ -11,9 +11,12 @@ convert_v2.py
 
 Training Notes:
 ```
-# dbrx trains like a much smaller model (~7B)
+# 1. dbrx trains like a much smaller model (~7B)
 # start with this as reference point and move up or down based on eval/train loss
 learning_rate = 1.5e-5  
+
+# 2. due to BPE (tiktoken) nature, tokenizer expansion/resize is not very friendly to training
+# use text based special tokens if you need/use extra tokens to avoid bad train/eval losses
 ```
 
 Known Issues: