How to calculate the number of invalid strings given a constraint system on alphabet, word blacklist, and string length
The language of valid strings is regular.  Build a DFA to recognize the language, then use standard techniques to count the number of words accepted by the DFA (see Why isn&#39;t it simple to count the number of words in a regular language? or https://cstheory.stackexchange.com/q/8200/5038).

How do you build a DFA to recognize the language?  Via a product construction.  Let $L_a$ denote the set of all words that don't have $n+1$ $a$'s in a row, and let $L_\rho$ denote the set of all words that aren't in the blacklist.  Then your language is $\Sigma^{\le m} \cap L_\rho \cap L_a \cap L_b \cap \cdots$.  You can easily build DFAs for each of the individual languages, and then use the product construction to build a DFA for their intersection.

Warning: the size of the DFA can potentially blow up exponentially in $|\Sigma|$.  However, this will remain polynomial in $n$, $m$, and $|\rho|$.