Pattern-matching function
I don't think that using pattern matching is necessary in this case, you only have a few simple conditions that can be better represented with a simple if. See what you think about this solution:

def getImportance(token: String, stopWords: Set[String]) =
  if (token.length() == 1 || token.length() &gt; 15 ||
    token.matches("\\p{Punct}+") ||
    stopWords.contains(token.toLowerCase()))
      0
  else
      1


I believe it is simpler and easier to read. I think using patter matching for these simple cases only hinders the understanding of the code ;)