Should I test my algorithm using a standard dataset that I believe is flawed?
You leave a lot unsaid here, such as how expensive (time, money) it is to run the algorithm on any given dataset and how much effort to create a new one. 

Three pieces of advice, though. If it is more or less standard to use the published data, then using that should give you some basis for evaluating your algorithm compared to others. With a new dataset you lose that opportunity. 

A new dataset also leaves you open to a charge that the "great" results you got were just an artifact of the dataset you created on your own. You should probably avoid that. At least as the first iteration. 

There is no reason that you couldn't use the existing dataset, gather your results, but also, in the same paper, discuss the shortcomings of that dataset. This also gives you an opening to future work and you might even get some advice on how to avoid the second point above. 

Ok. Four pieces of advice. If you are new at this (and not a doctoral candidate writing a dissertation) you don't need to push the boundaries too far to make a contribution. You can use the simplest case to set up your case 4. for the future. But if you buck current practice very much you need to justify it. 



On the other hand, if you are a doctoral candidate, the significance of your work is normally considered more important than the time you need to put into it and the expectation is that it is novel. But you still need to avoid the charge of tailoring the data to fit the desired results.