The following MWE example demonstrates the problem:

```
\documentclass{article}

\makeatletter
\def\arctg{\mathop{\operator@font ArcTg}\nolimits}
\makeatother

\usepackage{ifxetex,ifluatex}
\ifxetex
    \usepackage{fontspec}
    \setmainfont{Cambria}
    \usepackage{unicode-math}
    \setmathfont{Cambria Math}
\else\ifluatex
    \usepackage{fontspec}
    \defaultfontfeatures{Renderer=Basic,Ligatures={TeX}}
    \usepackage{unicode-math}
    \setmathfont{Cambria Math}
\else
    \usepackage[T1,T2A]{fontenc}
    \usepackage[utf8]{inputenc}
\fi\fi

\begin{document}
\ifxetex
    This is Xe\LaTeX
\else\ifluatex
    This is Lua\LaTeX
\else
    This is pdf\LaTeX
\fi\fi

 \subsubsection{Function $arctg(x)$}

 Function
 \begin{equation}
 \arctg(x)
 \end{equation}

 \subsubsection{Function $\cos(x)$}

 \begin{equation}
 \cos(x)
 \end{equation}

 \tableofcontents

\end{document}

```

I tried both Miktex and TL. At first run XeTeX produces `.toc` file with extremely long record: 

```
\contentsline {subsubsection}{\numberline {0.0.2}Function $\mathop {\Umathcode 97=7\symoperators 97\scan _stop: \Umathcode 98=7\symoperators 98\scan _stop:........

```

(here less than 1 percent is shown). At the second run xetex fails to finish the compilation. I did not chech if LuaLatex also fails. The compilation succeeds if `unicode-math` is not loaded.