Nullable nonterminals and recursion
A nonterminal $X$ is nullable if there is some derivation $X \to^* \epsilon$. It is clear that all non-terminals found by your algorithm are nullable (you can prove this by induction). On the other hand, if $X$ is nullable then take a derivation tree for the derivation, and prove by induction on the depth that all non-terminals on the way are recognized as nullable by this algorithm. This is why the algorithm is correct.

In your first example, $B$ is not nullable. In fact, you can't generate any string from $B$. The same is the case for your second example $S\to S$, where again the grammar generates no strings.