# Changelog

# 📢 Release v1.0.3
- 🚨 The `IndicProcessor` class has been re-written in [Cython](https://github.com/cython/cython) for faster implementation. This gives us atleast `+10 lines/s`.
- A new `visualize` argument as been added to `preprocess_batch` to track the processing with a `tqdm` bar.

# 📢 Release v1.0.2
- The repository has been renamed to `IndicTransToolkit`.
- 🚨 The custom tokenizer is now **removed** from the repository. Please revert to a previous commit ([v1.0.1](https://github.com/VarunGumma/IndicTransToolkit/tree/0e68fb5872f4d821578a5252f90ad43c9649370f)) to use it **(strongly discouraged)**. The official _(and only tokenizer)_ is available on HF along with the models.

# 📢 Release v1.0.0
- The [PreTrainedTokenizer](https://huggingface.co/docs/transformers/main_classes/tokenizer) for IndicTrans2 is now available on HF 🎉🎉 Note that, you still need the `IndicProcessor` to pre-process the sentences before tokenization.
- 🚨 **In favor of the standard PreTrainedTokenizer, we deprecated the custom tokenizer. However, this custom tokenizer will still be available here for backward compatibility, but no further updates/bug-fixes will be provided.**
- The `indic_evaluate` function is now consolidated into a concrete `IndicEvaluator` class.
- The data collation function for training is consolidated into a concrete `IndicDataCollator` class.
- A simple batching method is now available in the `IndicProcessor`.