Accuracy of word and sent tokenize versus custom tokenizers in nltk
Why would we want a custom tokenizer?  


Segementation is a very large topic, and as thus there is no perfect Natural Language Tokenizer.  Any toolkit needs to be flexible, and the ability to change the tokenizer, both so that someone can experiment, and so that it can be replaced if requirements are different, or better ways are found for specific problems, is useful and important.


  How could I measure if it was better that NLTK's tokenizer?


Anytime you are trying to quantify performance (ie: better) you will need to first define what is meant by better.  Once this is done, then typically you would perform the using the various methods under measurement, and then compare the results against your definition of better.  A couple of links which discuss these topics:  


Performance of Different NLP Toolkits in Formal and Social Text
Optimizing to Arbitrary NLP Metrics using Ensemble Selection