why does transform from tfidf vectorizer (sklearn) not work
As far as I can tell it's interpreting your new word_set word_set = ['dog', 'cat', 'foo'] as three separate documents containing one word each, whereas if you did word_set = ['dog cat foo'] it would interpret this as a single new document containing those words. 

What behavior are you expecting from this function? Is corpus = words a list of document strings, or a list of single words? If it's the latter, this is likely not doing what you think it is doing, and you should instead make corpus a list of document strings.