Statistical test for machine learning
I did something like that when I was at university. Professor asked me to implement and compare a variant of k-nearest neighbor with the original one.

I've applied the 10 fold cross validation over several datasets with both the algorithms and then I've applied the Friedman test with Holm post hoc over the accuracy means. You can use also simple paired t-test or others, it depends on your needs.

My professor also suggested me to use this graphical tool.
Hope this helps

SPOILER

It turned out that my k-NN variant wasn't better!