Added doc for setting tokenizers level of parallelism.
@@ -54,3 +54,10 @@ fn main() -> Result<()> {
Ok(())
}
```
+
+## Additional information
+
+- tokenizers is designed to leverage CPU parallelism when possible. The level of parallelism is determined
+by the total number of core/threads your CPU provides but this can be tuned by setting the `RAYON_RS_NUM_CPUS`
+environment variable. As an example setting `RAYON_RS_NUM_CPUS=4` will allocate a maximum of 4 threads.
+**_Please note this behavior may evolve in the future_**
