How to build a classifier for determining if two pairs are a match?
Generally your approach looks good to me. Here are a few comments/suggestions:


You didn't mention how the set of labelled pairs is obtained. This part can be tricky when doing record linkage among a large set of N addresses, since it's very difficult to manually annotate all the N*N pairs. Bootstrapping is a common approach afaik. 
The way the data is obtained may also have consequences about the proportion of positive/negative cases. In general the proportion positive cases is very low, and this can cause the usual problems of class imbalance. It's important not to train the model with a positive/negative ratio completely different from the one in the test set/in production.
It's often useful to diversify the similarity measure: instead of using only Jaro-Winkler, you can think of cosine-TFIDF, Levenshtein edit distance, and a lot of variants. In the same logic it's useful to vary the levels of comparison, e.g. characters, characters bigrams/trigrams, words.