What is ergodicity in a Markov Decision Process (MDP)?
In short, the relevant class of a MDPs that guarantees the existence of a unique stationary state distribution for every deterministic stationary policy are unichain MDPs (Puterman 1994, Sect. 8.3). However, the unichain assumption does not mean that every policy will eventually visit every state. I believe your confusion arises from the difference between unichain and more constrained ergodic MDPs.
Puterman defines that a MDP is (emphasis in the following mine):


recurrent or ergodic if the state transition matrix corresponding to every deterministic stationary policy consists of a single recurrent
class, and
unichain, if the state transition matrix corresponding to every deterministic stationary policy is unichain, that is, it consists of a
single recurrent class and a possibly empty set of transient states.


Before unpacking this further, let's first recap what recurrent and transient states in a Markov chain are. The following definitions can be found in Puterman, Appendix A.2. For a state $s$, associate two random variables $\nu_s$ and $\tau_s$ that represent the number of visits and the time of the first visit (or first return if the chain starts in $s$) to state $s$. A state $s$ is

recurrent (sometimes also called ergodic), if $P_s(\tau_s &lt; \infty) = 1$, that is, the state is eventually visited (or returned to), and
transient, if $\mathbb{E}[\tau_s] = \infty$ and $P_s(\tau_s &lt; \infty) &lt; 1$.

It is also true that $s$ is recurrent if and only if $\mathbb{E}[\nu_s] = \infty$, i.e., it is visited infinitely often, and $s$ is transient if and only if $\mathbb{E}[\nu_s] &lt; \infty$, i.e., it is visited only finitely often (and thereby never again after some finite time).
So let's now return to the two types of MDPs above. Consider an arbitrary deterministic stationary policy $\pi$ which maps any state $s$ to an action $a = \pi(s)$. If the MDP is ergodic, then the stationary distribution $\mu_\pi(s)$ exists and is unique, because the Markov chain over states induced by any policy has a single recurrent class (it does not matter in which state $s_0$ the chain starts, the same stationary distribution is reached). There is a single class of recurrent states, i.e., all states are recurrent, therefore any $s$ is visited infinitely often and $\mu_\pi(s) &gt; 0$.
Now, if the MDP is unichain, then once again the stationary distribution $\mu_\pi(s)$ exists and is unique, because the Markov chain over states induced by any policy has a single recurrent class. But, importantly, there may exist a policy $\pi$ for which the set of transient states in the induced Markov chain is not empty. Because any transient state $s$ will only be visited a finite number of times, in the (infinite horizon) stationary distribution $\mu_\pi(s)=0$!
So indeed, if the MDP is of the stricter ergodic type, every policy will eventually visit every state. This is not true for unichain MDPs however.
A final remark: some authors define a policy as ergodic (e.g., Kearns &amp; Singh, 2002), if the resulting Markov chain over states is ergodic (i.e., has a unique stationary distribution). The unichain MDP is a type of MDP where every policy is ergodic.
References:
Puterman, M. L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming.
Kearns &amp; Singh. Near-Optimal Reinforcement Learning in Polynomial Time. Machine Learning, 49, 209–232, 2002