How do ion channels transport only specific ions?
I am restricting the answer to only $Na^+$ and $K^+$ channels, assuming similar mechanism for other channels. In these 2 channels, such high level of specificity is achieved because of two main differences between $Na^+$ and $K^+$:


difference in size of $Na^+$ and $K^+$ ions.
difference in chemical properties of $Na^+$ and $K^+$ ions.


Lets discuss these points one-by-one:


Difference in ionic size: the ionic size of $Na^+$ ion is 0.96$\overset{\circ}{A}$ while that of $K^+$ ion is 1.33$\overset{\circ}{A}$. How this size difference of 0.38$\overset {\circ}{A} $ is exploited by channels will be discussed in later section.


source


Difference in chemical properties: the property we are concerned here with is hydration enthalpy. $Na^+$ ion has hydration enthalpy than $K^+$ ion, meaning $Na^+$ has higher tendency to remain bounded by water molecules than $K^+$. How this enthalpy difference is exploited by channels will be discussed in next section.

source


I will discuss different methods used by these channels one at a time.


$Na^+$ channels: $Na^+$ channels are less studied than $K^+$ channels, but some of the mechanisms used are known and explained below:


$Na^+$ ions pass through channels in partially hydrated form1, and these channels have 2 rings of selection filters2 in mammals.
The first (outer) ring contains amino acids EEDD (or EEMD)3 while second (inner) ring contains amino acids DEKA4. How these amino acids bind to specifically $Na^+$ ions is not completely known yet, but it is hypothesized that the inner ring, being more conserved, is more important as a filter than the outer ring. Also, it has been shown that by changing the amino acid K in DEKA makes the channel more permeable to $K^+$ and $Ca^{2+}$ ions, and by changing sequence from DEKA to EEEE makes the channel more selective towards $Ca^{2+}$ ions5.
$K^+$ ion, in hydrated form, cannot pass through inner filter due to larger size and presence of negatively charged D and E of DEKA, whereas $Na^+$ easily passes through due to smaller size and higher hydration enthalpy6.





Negative charge in channel is such that it mimics the hydration shell (regular octahedron) of $Na^+$ from side

source (1)           source (2)


$K^+$ channels: $K^+$ channels have been studied more intensively, so their mechanism is much more understood. The main points to ponder are:


The filter of $K^+$ channel contains amino acids TVGYG 7 and is present in the narrowest region of the channel.
To pass this channel, $K^+$ ion needs to be completely dehydrated i.e. without any water molecule bound to it. The amino acids in the filter replace water molecules with polar oxygen atoms. On the other hand, $Na^+$ cannot get rid of water molecules so easily and hence, its passage through the channel is thermodynamically unfavorable8, partly also because dehydrated $Na^+$ is so small that it cannot even bind with all of the polar oxygen atoms of amino acids.
Here nature shows some clever techniques. The amino acids, TVGYG, having their electronegative carbonyl oxygen atoms aligned toward the center of the filter pore, form a square anti-prism similar to a water-solvating shell around each potassium binding site. The distance between the carbonyl oxygens and potassium ions in the binding sites of the selectivity filter is the same as between water oxygens in the first hydration shell and a potassium ion in water solution, providing an energetically-favorable route for de-solvation of the ions9.



This is how the passage of a $K^+$ ion through $K^+$ channel looks like:
source

The carbonyl oxygens mimic this configuration as this is how a $K^+$ looks in hydrated form (square antiprism):source

P.S. as more deep studies are made, more mechanisms may be revealed. Also, above mentioned points are just theories and not conclusive proofs.

References:


Hille, B. The permeability of the sodium channel to organic cations in myelinated nerve. J. Gen. Physiol. 58, 599–619 (1971).
Guy HR, Seetharamulu P. Molecular-Model of the Action-Potential Sodium-Channel. Proc Natl Acad Sci USA. 1986;83:508–512.
McNulty, M. M., Edgerton, G. B., Shah, R. D., Hanck, D. A., Fozzard, H. A. and Lipkind, G. M. (2007), Charge at the lidocaine binding site residue Phe-1759 affects permeation in human cardiac voltage-gated sodium channels. The Journal of Physiology, 581: 741–755. doi:10.1113/jphysiol.2007.130161
Favre I., Moczydlowski E. &amp; Schild L. On the structural basis for ionic selectivity among Na+, K+, and Ca2+ in the voltage-gated sodium channel. Biophys J 71, 3110–3125 (1996).
Calcium channel characteristics conferred on the sodium channel by single mutations.
Heinemann SH, Terlau H, Stühmer W, Imoto K, Numa S 1992 Apr 2;356(6368):441-3.
Mechanism of Ion Permeation and Selectivity in a Voltage Gated Sodium Channel. Ben Corry and Michael Thomas
Chemistry of ion coordination and hydration revealed by a $K^+$ channel–Fab complex at 2.0 Å resolution, Yufeng Zhou, João H. Morais-Cabral, Amelia Kaufman &amp; Roderick MacKinnon
Mechanism of potassium channel selectivity revealed by $Na^+$ and $Li^+$ binding sites within the KcsA pore.
Ameer N. Thompson, Ilsoo Kim, Timothy D. Panosian, Tina M. Iverson, Toby W. Allen, and Crina M. Nimigean
A comparison between two prokaryotic potassium channels (KirBac1.1 and KcsA) in a molecular dynamics (MD) simulation study; Mikko Hellgrena, Lars Sandberg, Olle Edholm, 1 March 2006