How to compare a machine learning model and a rule based model
For the rule base model I would take the exact same approach as for ML model, split the data into train and test set since at the end you want to check whether or not your rule base and ml models generalize well
It is also recommended to used the same sets to train and evaluate both models in such a way that it is a fair comparison.