Classifier runtime evaluation
I don't have much hands-on experience in this, but nevertheless I think this kind of processing of streamingdata is called "online learning"; and finding optimal hyperparameters for any type of learning (online or batch) automatically such that the system is self-tuning is an area of active research. 

Prof. George Hinton talks about it generally in this video "Lecture 16C : Bayesian optimization of neural network hyperparameters" in this playlist: https://www.youtube.com/playlist?list=PLiPvV5TNogxKKwvKb1RKwkq2hm7ZvpHz0
(13min) 

If you like audio, a listener asked a similar question in this  Episode of the Talking Machines podcast: 
(about 8 minutes in)

Clever switching of the learning algorithm also has been explored, see "Bagging" or "Boosting" on Wikipedia, or Ensemble learning

For a relatively easy-to-use Java-based GUI tool you might try "MOA"- Massive ONline Analysis