The meaning of permanent dipole moment
Indeed, $\ce{XeF6}$ has a distorted tetrahedral structure in all three phases. Two of solid state crystal structure modifications ($\bf{A}$ and $\bf{B}$) are shown in following diagram (Ref.1):



The abstract of Ref.1 tells them all:


  According to single crystal X-ray diffraction, neutron powder diffraction, solid state MAS NMR data, and differential scanning calorimetry, $\ce{XeF6}$ exists in at least six different modifications. Three of them are formed at temperatures above room temperature, one exists at room temperature, while two have been found at low temperatures. In the high temperature modifications $\ce{XeF6}$ forms a non-symmetric tetramer, better described as a cyclic trimer with a weakly associated monomer. The normal temperature modification is the previously described cubic phase IV, having disordered tetrameric and hexameric units. The low temperature modifications are regular tetramers. Only in presence of $\ce{HF}$ symmetric dimers are formed.


The tetramer $\bf{C}$ (ball and stick version of $\bf{A}$) represent the general version of all modifications, a cyclic trimer with a weakly associated monomer. The symmetric dimer $\bf{D}$ is the structure of $\ce{XeF6}$ in the presence of $\ce{HF}$ (highlighted by $\color{green}{\text{green}}$ ovals). Nonetheless, all of these modifications display distorted octahedral structure for individual $\ce{XeF6}$ molecule as represented by ball &amp; stick 3D structure in the central box. None of $\ce{F-Xe-F}$ angles are equal to $90^\circ$, which is the normal angle of central atom with two adjacent ligands in regular octahedron. As a consequence, $\ce{XeF6}$ has net permanent dipole moment since all six $\ce{Xe^\delta+-F^\delta-}$ bonds are polar.

For your question about ammonia, look at Ivan Neretin's comment and here where there is an accepted answer.

References:


Sevim Hoyer, Thomas Emmler, Konrad Seppelt, “The structure of xenon hexafluoride in the solid state,” Journal of Fluorine Chemistry 2006, 127(10), 1415-1422 (https://doi.org/10.1016/j.jfluchem.2006.04.014).