If a language has any single occurrence of a letter, is it not context-free?
No: context free languages are closed under union, so for any CFL language $L$, we have $L \cup \{ a \}$ is context free, for any letter $a$.

The trick is that, when applying the pumping lemma, we don't get to choose the length $L$.

For a finite language, you just choose $L$ to be larger than any word in the language, and the pumping lemma trivially holds. (Remember, $a \implies b$ is true whenever $a$ is false).

As an intuition, you can see the "length" part of the lemma as being how long of words we need to get to a part of the automaton with loops, or a part of the grammar with recursion. So the small examples will in general not affect the pumping property.