Note to use dbrx-base-tokenizer

README.md

@@ -16,9 +16,10 @@ Training Notes/Observations:
 # start with this as reference point and move up or down based on eval/train loss
 learning_rate = 1.5e-5  
 ```
-2. Due to nature of BPE (tiktoken), tokenizer expansion/resize is not very friendly to training. Use text based special tokens if you need/use extra tokens to avoid bad train/eval losses
+2. Highly recommend to train this model with `dbrx-base-tokenizer` tokenizer (fully-compatible): https://huggingface.co/LnL-AI/dbrx-base-tokenizer
 
-Quants:
+
+# Quants:
 
 1. 4bit gptq/marlin: https://huggingface.co/LnL-AI/dbrx-base-converted-v2-4bit-gptq-marlin
 2. 4bit gptq/gptq: https://huggingface.co/LnL-AI/dbrx-base-converted-v2-4bit-gptq-gptq