website or service using algorithm awhile the rest of the users will be provided with algorithm for both groupsrelevant success metrics will be recorded for set period of time thenthe metrics of algorithm and algorithm will be comparedand selection between the two approaches will be made according to these metrics using / testing enables us to evaluate the algorithms "in the wild,which might help us to discover unexpected consequences when users are interacting with our model often is new modelwhile is the established system there are more elaborate mechanisms for online testing that go beyond / testingsuch as bandit algorithms great introduction to this subject can be found in the book bandit algorithms for website optimization by john myles white ( 'reillybuilding your own estimator this book has covered variety of tools and algorithms implemented in scikitlearn that can be used on wide range of tasks howeveroften there will be some particular processing you need to do for your data that is not implemented in scikit-learn it may be enough to just preprocess your data before passing it to your scikit-learn model or pipeline howeverif your preprocessing is data dependentand you want to apply grid search or cross-validationthings become trickier in we discussed the importance of putting all data-dependent processing inside the cross-validation loop so how can you use your own processing together with the scikit-learn toolsthere is simple solutionbuild your own estimatorimplementing an estimator that is compatible with the scikit-learn interfaceso that it can be used with pipelinegridsearchcvand cross_val_scoreis quite easy you can find detailed instructions in the scikit-learn documentationbut here is the gist the simplest way to implement transformer class is by inheriting from baseestimator and transformermixinand then implementing the __init__fitand predict functions like this wrapping up