Explicit Multi-point Taylor Polynomial

Abstract: The multi-point Taylor polynomial, which is the general, unique and of minimum degree ($mk+m-1$) polynomial $P_{k,m}(x)$ which interpolates a function's derivatives in multiple points is presented in its explicit form. A proof that this expression satisfies the multi-point Taylor polynomial's defining property is given. Namely, it is proven that for a k-differentiable function $f$ and a set of different m-points $\{a_1,...,a_m\}$, this polynomial satisfies $P^{(n)}_{k,m}(a_i) = f^{(n)}(a_i)\quad \forall \, i = 1,...,m\quad \&\quad \forall \, n = 0,...,k$. A discussion regarding previous expressions presented in the literature, which mostly consisted in recursion formulas and not explicit formulas, is made.

\section{Introduction}

It is well known that an infinitely-differentiable function $f:I\longrightarrow \mathbb{R}$ can represented by its Taylor series expansion around a single point $a\in I$. This in turn defines the single-point Taylor polynomial (STP) as the partial sum of the series, given by
\begin{equation}
\label{Taylor}
    p_k(x) = \sum_{n=0}^{k}\frac{(x-a)^{n}}{n!}f^{(n)}(a),
\end{equation}
which produces a good approximation of the function for increasing values of $k$ and in fact exactly interpolates the function's derivatives at the point $a$. That is, the STP is the unique and of minimal (of order $k$) polynomial with the property that
\begin{equation}
    p^{(n)}_k(a) = f^{(n)}(a) \quad \forall\, n = 0,...,k.
\end{equation}
Furthermore, the STP can be defined for functions that are just $k$-differentiable, which together with the previous properties makes it very suitable for polynomial interpolation.

 On the other hand, for a set of different points $\mathcal{A} = \{a_1,...,a_m\}\subset I$, the Lagrange polynomial (LP)
\begin{equation}
\label{Lagrange}
    q_{m}(x) = \sum_{g=1}^{m}\prod^{m}_{\substack{h=1\\
    h\neq g}}\frac{(x-a_h)}{(a_g-a_h)}f(a_g),
\end{equation}
gives the unique polynomial (of minimum order $m$)  that interpolates the value of the function at the given points
\begin{equation}
    q_m(a_i) = f(a_i) \quad \forall\, i = 1,...,m,
\end{equation}
and can be found as a basic multi-point interpolating polynomial in many numerical analysis textbooks~\cite{Richard,Philip,Bulirsch,Hildebrand}.

One can go further and ask for a polynomial which combines both of these properties, defining the multi-point Taylor polynomial (MTP). That is, for a k-differentiable real function $f:I\longrightarrow \mathbb{R}$ and a set of different points $\mathcal{A} = \{a_1,...,a_m\}\subset I$, the MTP is the unique and minimal polynomial (of order $mk+m-1$) which fulfills the conditions
\begin{equation}
\label{property}
    P^{(n)}_{k,m}(a_i) = f^{(n)}(a_i)\quad \forall \, i = 1,...,m\quad \&\quad \forall \, n = 0,...,k.
\end{equation}
The solution to these conditions is also known as the Hermite or Osculatory interpolation polynomial, and multiple standard methods and iterative algorithms can be found for producing it~\cite{Richard,Philip,Bulirsch,Note_2Point}, but these do not give its explicit and final form. This is what will be presented on this paper.

The use of MTP or similar interpolants has encountered many applications. For example, the authors in Ref.~\cite{NASA} presented a two-point Taylor series expansion whose coefficients can be iterated to produce higher order expressions and applied this to the two-body problem, which consisted of an expansion of the solutions both at the perigee and apogee. Their results turned out to be significantly better than a basic Taylor approximation around just the perigee. Another study~\cite{Lopez} used a convenient form of the MTP and applied it to the approximation of solutions of second order linear differential equations on an interval with boundary conditions on the extremes, which was done by approximating the solutions around the two boundary points. On the other hand, the study~\cite{Diff_eq} performed a multiple point expansion by requesting the satisfaction of the boundary conditions and of the differential equation at a cloud of points inside the domain. Finally, the work in~\cite{Image_comp} used a multi-point Taylor series formula in terms of general basis functions in order to construct an image compression/decompression method.

\section{The multi-point Taylor polynomial}

The explicit form of the MTP will now presented in the form of a theorem and is the main result of this paper.

\textbf{Theorem:}

For a k-differentiable real function $f:I\longrightarrow \mathbb{R}$ and a set of different points $\mathcal{A} = \{a_1,...,a_m\}\subset I$, the explicit expression for the MTP is
\begin{equation}
\label{Polynomial}
    P_{k,m}(x) = \sum_{g=1}^{m}\bigg[\prod^{m}_{\substack{h=1\\
    h\neq g}}\frac{(x-a_h)}{(a_g-a_h)}\bigg]^{k+1}\sum_{n=0}^{k}\frac{(x-a_g)^{n}}{n!}F^{n,g}_{k,m}[\mathcal{A}],
\end{equation}
where $F^{n,g}_{k,m}[\mathcal{A}]$ are constant factors given by
\begin{equation}
    F^{n,g}_{k,m}[\mathcal{A}] = \sum_{j_1+...+j_m = n}\binom{n}{j_1,...,j_m}f^{(j_g)}(a_g)\prod^{m}_{\substack{l=1\\
    l\neq g}}\frac{(k+j_l)!}{k!}(a_l-a_g)^{-j_l}.
\end{equation}
This polynomial satisfies the conditions (\ref{property}) and thus interpolates a function's derivatives at multiple points.

Under the understanding that empty sums are null and empty products are one, this expression recovers the STP of Eq. (\ref{Taylor}) for $m = 1$, while $k=0$  recovers the LP of Eq. (\ref{Lagrange}). This is a straightforward property which comes from the fact that the conditions (\ref{property}) recover the STP's or the LP's conditions respectively for each case.

\textbf{Proof:}

It will now be proven that the polynomial $P_{k,m}(x)$ of Eq. (\ref{Polynomial}) satisfies the conditions (\ref{property}). First notice that $P^{(n)}_{k,m}(a_i)$ will only have contributions from the $g=i$ term of the first sum, as the order of the derivative is not big enough to suppress the $(x-a_i)$ factor involved in the other terms which involve
\begin{equation}
    \bigg[\prod^{m}_{\substack{h=1\\
    h\neq g}}\frac{(x-a_h)}{(a_g-a_h)}\bigg]^{k+1},
\end{equation}
with $g\neq i$, as it is assumed that $n<k+1$. Therefore,

\begin{equation}
\label{a}
\begin{split}
    P^{(n)}_{k,m}(a_i) = \frac{\mathrm{d}^{n}}{\mathrm{d}x^{n}}\Bigg[\bigg[\prod^{m}_{\substack{h=1\\
    h\neq i}}\frac{(x-a_h)}{(a_i-a_h)}\bigg]^{k+1}\sum_{s=0}^{k}&\frac{(x-a_i)^{s}}{s!}F^{s,i}_{k,m}[\mathcal{A}]\Bigg]_{x=a_i}. \\
\end{split}
\end{equation}

With that, using the well known Leibniz rule for higher order derivatives
\begin{equation}
    (fg)^{(n)} = \sum_{r=0}^{n}\binom{n}{r}f^{(n-r)}g^{(r)},
\end{equation}
then
\begin{equation}
\label{b}
\begin{split}
\frac{\mathrm{d}^{n}}{\mathrm{d}x^{n}}\Bigg[\bigg[\prod^{m}_{\substack{h=1\\
    h\neq i}}\frac{(x-a_h)}{(a_i-a_h)}\bigg]^{k+1}\sum_{s=0}^{k}\frac{(x-a_i)^{s}}{s!}A_s\Bigg]_{x=a_i} & =  \sum_{r=0}^{n}\binom{n}{r}\frac{\mathrm{d}^{r}}{\mathrm{d}x^{r}}\Bigg[\bigg[\prod^{m}_{\substack{h=1\\
    h\neq i}}\frac{(x-a_h)}{(a_i-a_h)}\bigg]^{k+1}\Bigg]_{x=a_i} \\
    &\hspace{4cm}\times \sum_{s=0}^{k}\frac{(a_i-a_i)^{s-n+r}}{(s-n+r)!}F^{s,i}_{k,m}[\mathcal{A}]. \\
\end{split}
\end{equation}

Again, only the $s = n-r$ term survives. Now, the general Leibniz rule for more than two factors is
\begin{equation}
    \bigg(\prod_{h=1}^{m} f_h\bigg)^{(n)} = \sum_{k_1+...+k_m = n}\binom{n}{k_1,...,k_m}\prod_{h=1}^{m} f_h^{(k_h)}
\end{equation}
(a proof for this relation can be found for example in Ref. \cite{Gen_Leibniz}), so
\begin{equation}
\begin{split}
    \frac{\mathrm{d}^{r}}{\mathrm{d}x^{r}}\Bigg[\prod^{m}_{\substack{h=1\\
    h\neq i}}\frac{(x-a_h)^{k+1}}{(a_i-a_h)^{k+1}}\Bigg]_{x=a_i} & = \sum_{k_1+...+k_m = r}^{i}\binom{r}{k_1,...,k_m}^i\prod_{\substack{h=1\\
    h\neq i}}^{m} \frac{(k+1)!}{(k+1-k_h)!}\frac{(a_i-a_h)^{k+1-k_h}}{(a_i-a_h)^{k+1}} \\
    & = [(k+1)!]^{m-1}\sum_{k_1+...+k_m = r}^{i}\binom{r}{k_1,...,k_m}^i\prod_{\substack{h=1\\
    h\neq i}}^{m} \frac{1}{(k+1-k_h)!}(a_i-a_h)^{-k_h},
\end{split}
\end{equation}
where the multinomial sum and multinomial coefficient with upper index $\sum_{k_1+...+k_m = r}^{i}$ and $\binom{r}{k_1,...,k_m}^i$ denote that $k_i$ is not taken into account. Joining this with equations (\ref{a}) and (\ref{b}), it follows that

\begin{equation}
\begin{split}
P^{(n)}_{k,m}(a_i) & =  \sum_{r=0}^{n}\binom{n}{r} [(k+1)!]^{m-1}F^{n-r,i}_{k,m}[\mathcal{A}]\sum_{k_1+...+k_m = r}^{i}\binom{r}{k_1,...,k_m}^i\prod_{\substack{h=1\\
    h\neq i}}^{m} \frac{1}{(k+1-k_h)!}(a_i-a_h)^{-k_h}. \\
\end{split}
\end{equation}
Now,
\begin{equation}
\begin{split}
    \binom{n}{r}[(k+1)!]^{m-1}\binom{r}{k_1,...,k_m}^i&F^{n-r,i}_{k,m}[\mathcal{A}] \\
    & = \frac{(k+1)^{m-1}}{(n-r)!}\binom{n}{k_1,...,k_m}^i\sum_{j_1+...+j_m = n-r}\binom{n-r}{j_1,...,j_m}f^{(j_i)}(a_i)\prod^{m}_{\substack{l=1\\
    l\neq i}}\frac{(k+j_l)!}{(a_l-a_i)^{j_l}} \\
    & = (k+1)^{m-1}\binom{n}{k_1,...,k_m}^i\sum_{j_1+...+j_m = n-r}\binom{1}{j_1,...,j_m}f^{(j_i)}(a_i)\prod^{m}_{\substack{l=1\\
    l\neq i}}\frac{(k+j_l)!}{(a_l-a_i)^{j_l}}. \\
\end{split}
\end{equation}
Therefore, the evaluation of the polynomial can be expressed as the sum
\begin{equation}
\label{c}
\begin{split}
P^{(n)}_{k,m}(a_i) & =  (k+1)^{m-1}\sum_{r=0}^{n}B_r, \\
\end{split}
\end{equation}
with
\begin{equation}
    B_r = \sum_{k_1+...+k_m = r}^{i}\binom{n}{k_1,...,k_m}^i\sum_{j_1+...+j_m = n-r}\binom{1}{j_1,...,j_m}f^{(j_i)}(a_i)\prod^{m}_{\substack{l=1\\
    l\neq i}}\prod_{\substack{h=1\\
    h\neq i}}^{m}\frac{(k+j_l)!}{(k+1-k_h)!(a_i-a_h)^{k_h}(a_l-a_i)^{j_l}}.
\end{equation}

As the products range between the same indices, these can be joined together under the same label. Furthermore, when grouping the last two denominators, a global sign appears, which is given by
\begin{equation}
    \prod_{\substack{h=1\\
    h\neq i}}^{m}(-1)^{k_h} = (-1)^{r},
\end{equation}
because $k_1+...+k_m = r$, where $k_i$ is not considered. The result of this process is
\begin{equation}
    B_r = (-1)^{r}\sum_{k_1+...+k_m = r}^{i}\binom{n}{k_1,...,k_m}^i\sum_{j_1+...+j_m = n-r}\binom{1}{j_1,...,j_m}f^{(j_i)}(a_i)\prod_{\substack{h=1\\
    h\neq i}}^{m}\frac{(k+j_h)!}{(k+1-k_h)!}(a_h-a_i)^{-j_h-k_h}.
\end{equation}

The only term that could produce the case $j_i = n$, so that the expression includes a $f^{(n)}(a_i)$ term, is the one where $r=0$. Making an explicit calculation of this term,
\begin{equation}
\begin{split}
    B_0 & = (-1)^{0}\sum_{k_1+...+k_m = 0}^{i}\binom{n}{k_1,...,k_m}^i\sum_{j_1+...+j_m = n}\binom{1}{j_1,...,j_m}f^{(j_i)}(a_i)\prod_{\substack{h=1\\
    h\neq i}}^{m}\frac{(k+j_h)!}{(k+1-k_h)!}(a_h-a_i)^{-j_h-k_h} \\
    & = \sum_{j_1+...+j_m = n}\binom{n}{j_1,...,j_m}f^{(j_i)}(a_i)\prod_{\substack{h=1\\
    h\neq i}}^{m}\frac{(k+j_h)!}{(k+1)!}(a_h-a_i)^{-j_h}. \\
\end{split}
\end{equation}
Separating the sought $j_i = n$ term from the rest,
\begin{equation}
\begin{split}
    B_0 & = f^{(n)}(a_i)\prod_{\substack{h=1\\
    h\neq i}}^{m}\frac{k!}{(k+1)!}(a_h-a_i)^{0} + \sum_{j_1+...+j_m = n}^{j_i\neq n}\binom{n}{j_1,...,j_m}f^{(j_i)}(a_i)\prod_{\substack{h=1\\
    h\neq i}}^{m}\frac{(k+j_h)!}{(k+1)!}(a_h-a_i)^{-j_h} \\
    & = \frac{1}{(k+1)^{m-1}}f^{(n)}(a_i)+ \sum_{j_1+...+j_m = n}^{j_i\neq n}\binom{n}{j_1,...,j_m}f^{(j_i)}(a_i)\prod_{\substack{h=1\\
    h\neq i}}^{m}\frac{(k+j_h)!}{(k+1)!}(a_h-a_i)^{-j_h}, \\
\end{split}
\end{equation}
which gives the right term when inserted in equation (\ref{c}). Thus, now it must only be proven that all the other terms cancel each other. For this, it is convenient to group every coefficient of the derivatives of $f(a_i)$. That is,
\begin{equation}
\label{Expansion}
\begin{split}
    P^{(n)}_{k,m}(a_i) & =  (k+1)^{m-1}\sum_{r=0}^{n}B_r =  f^{(n)}(a_i)+n!\sum_{j=0}^{n-1}\frac{f^{(j)}(a_i)}{(j)!}C_{j},
\end{split}
\end{equation}
where, for $0\leq j \leq n-1$,
\begin{equation}
\begin{split}
    C_{j} & = \frac{(k+1)^{m-1}}{n!}\sum_{r=0}^{n-j}(-1)^{r}\sum_{k_1+...+k_m = r}^{i}\binom{n}{k_1,...,k_m} \\
    &\hspace{6cm}\times\sum_{j_1+...+j_m = n-r-j}^{i}\binom{1}{j_1,...,j_m}\prod_{\substack{h=1\\
    h\neq i}}^{m}\frac{(k+j_h)!}{(k+1-k_h)!}(a_h-a_i)^{-j_h-k_h} \\
    & =(k+1)^{m-1}\sum_{r=0}^{n-j}(-1)^{r}\sum_{\substack{k_1+...+k_m = r \\
    j_1+...+j_m = n-r-j}}^{i}\prod_{\substack{h=1\\
    h\neq i}}^{m}\frac{(k+j_h)!}{(k+1-k_h)!k_h!j_h!}(a_h-a_i)^{-j_h-k_h} \\
    & = \sum_{r=0}^{n-j}(-1)^{r}\sum_{\substack{k_1+...+k_m = r \\
    j_1+...+j_m = n-r-j}}^{i}\prod_{\substack{h=1\\
    h\neq i}}^{m}\binom{k+j_h}{j_h}\binom{k+1}{k_h}(a_h-a_i)^{-j_h-k_h}. \\
\end{split}
\end{equation}

These terms automatically include the $j_i\neq n$ term from $B_0$, whose nullity was left pendent to be proven. The restricted limit on the sum over $r$ comes from the fact that $j> n-r$ is not taken. The notation $\sum_{j_1+...+j_m = n-r-j}^{i}$ is used again to denote that $j_i$ is not considered (in fact $j_i = j$). This applies for the sum over $k_i$ as well. The task is now to prove that each factor $C_j$ is equal to zero.

Now, because
\begin{equation}
    \sum_{r=0}^{n-j}(-1)^{r}\sum_{\substack{k_1+...+k_m = r \\
    j_1+...+j_m = n-r-j}}^{i}\big[...\big] = \sum_{S(k_h)+S(j_h) = n-j}^{i}(-1)^{S(k_h)}\big[...\big],
\end{equation}
where $S(k_h) = k_1+...+k_m$ and $S(j_h) = j_1+...+j_m$ (without $k_i$ nor $j_i$), it follows that
\begin{equation}
\begin{split}
    C_{j} & = \sum_{S(k_h)+S(j_h) =n_j}^{i}(-1)^{S(k_h)}\prod_{\substack{h=1\\
    h\neq i}}^{m}\binom{k+j_h}{j_h}\binom{k+1}{k_h}(a_h-a_i)^{-j_h-k_h}, \\
\end{split}
\end{equation}
where the positive integer $n_j = n-j$ was defined. It is now convenient to further join together the terms of the same power over $(a_h-a_i)$. That is, the terms with $j_w+k_w = s_w$, for $w = 1,...,m$ and $w\neq i$ will now be grouped, which is done by performing the sum over the possible $s_w$ terms. This results in
\begin{equation}
\begin{split}
    C_{j} & = \sum_{s_1+...+s_m = n_j}^{i} \sum_{\substack{j_v+k_v=s_v\\
    v=1,...,m}}^{i}\prod_{\substack{h=1\\
    h\neq i}}^{m}(-1)^{k_h}\binom{k+j_h}{j_h}\binom{k+1}{k_h}(a_h-a_i)^{-j_h-k_h} \\
    & = \sum_{s_1+...+s_m = n_j}^{i}\bigg[\prod_{\substack{w=1\\
    w\neq i}}^{m}(a_w-a_i)^{-s_w}\bigg] \sum_{\substack{j_v+k_v=s_v\\
    v=1,...,m}}^{i}\prod_{\substack{h=1\\
    h\neq i}}^{m}(-1)^{k_h}\binom{k+j_h}{j_h}\binom{k+1}{k_h}. \\
\end{split}
\end{equation}
Once again, the upper index on the sum denotes that the respective term with sub-index equal to $i$ is not considered. Having grouped every non-compatible term, it is now left to be proven that the factor of each one is zero. That is, one must prove that, for any fixed combination $\mathcal{S} = \{s_1,...,s_m\}$ such that $s_1+...+s_m = n_j$ (without considering $s_i$), the term
\begin{equation}
    D_{n_j,k}(\mathcal{S}) \equiv \sum_{\substack{j_v+k_v=s_v\\
    v=1,...,m}}^{i}\prod_{\substack{h=1\\
    h\neq i}}^{m}(-1)^{k_h}\binom{k+j_h}{j_h}\binom{k+1}{k_h}
\end{equation}
is identically zero, where $1 \leq n_j \leq n \leq k$. This will be done by strong induction over $n_j$. For $n_j=1$ the only possible combination is when only one element of $\mathcal{S}$ is equal to one, while the others are zero. Without loss of generality, this element can be taken to be $s_1$. Then, the sum
\begin{equation}
    \sum_{\substack{j_v+k_v=s_v\\
    v=1,...,m}}^{i}
\end{equation}
has two terms: One where $(j_1,k_1)=(1,0)$ and another where $(j_1,k_1)=(0,1)$, where in both cases all of the other $(j_v,k_v)$ pairs are zero. Thus
\begin{equation}
\begin{split}
    D_{1,k}(\mathcal{S}) & = \bigg[(-1)^{0}\binom{k+1}{1}\binom{k+1}{0}+(-1)^{1}\binom{k+0}{0}\binom{k+1}{1}\bigg]\prod_{\substack{h=2\\
    h\neq i}}^{m}(-1)^{0}\binom{k+0}{0}\binom{k+1}{0} \\
    & = (k+1)-(k+1) \\
    & = 0.
\end{split}
\end{equation}
Now the inductive step will be performed. Suppose that, for any integer $n'$ such that $1\leq n'\leq n_j$, any combination $\mathcal{S} = \{s_1,...,s_m\}$ such that $s_1+...+s_m = n'$ (without considering $s_i$) produces a $D_{n',k}(\mathcal{S})$ term which vanishes. Take $n_j+1$ and an arbitrary combination $\mathcal{S}' = \{s'_1,...,s'_m\}$ such that $s'_1+...+s'_m = n_j+1$ (without considering $s'_i$). Now it must be proven that $D_{n_j+1,k}(\mathcal{S}')$ is zero.

It is clear that there exists a combination $\mathcal{S} = \{s_1,...,s_m\}$ such that $s_1+...+s_m = n_j$ and all of the elements of $\mathcal{S}$ are equal to the ones of $\mathcal{S}'$ with the exception of one, which differs by one unity. This can be done by just taking a copy of $\mathcal{S}'$ except for one element $s'$ greater than zero (which always exists as $n'\geq 1$), for which the element $s'-1$ is taken instead. Without loss of generality, this element can be taken as the first one. Thus,
\begin{equation}
    \mathcal{S}' = \{s_1+1,s_2,...,s_m\},
\end{equation}
and therefore
\begin{equation}
\begin{split}
    D_{n_j+1,k}(\mathcal{S}') & = \sum_{j_1+k_1=s_1+1}\sum_{\substack{j_v+k_v=s_v\\
    v=2,...,m}}^{i}\prod_{\substack{h=1\\
    h\neq i}}^{m}(-1)^{k_h}\binom{k+j_h}{j_h}\binom{k+1}{k_h} \\
    & = \sum_{j_1+k_1=s_1+1}(-1)^{k_1}\binom{k+j_1}{j_1}\binom{k+1}{k_1}\sum_{\substack{j_v+k_v=s_v\\
    v=2,...,m}}^{i}\prod_{\substack{h=2\\
    h\neq i}}^{m}(-1)^{k_h}\binom{k+j_h}{j_h}\binom{k+1}{k_h}. \\
\end{split}
\end{equation}
The second sum of this equation can be expressed as a particular case of $D$. That is, if one takes the combination $\mathcal{S}'' = \{0,s_2...,s_m\}$, then $0+s_2+...+s_m = n_j-s_1$ and thus
\begin{equation}
\begin{split}
    D_{n_j-s_1,k}(\mathcal{S}'') & = (-1)^0\binom{k+0}{0}\binom{k+1}{0}\sum_{\substack{j_v+k_v=s_v\\
    v=2,...,m}}^{i}\prod_{\substack{h=2\\
    h\neq i}}^{m}(-1)^{k_h}\binom{k+j_h}{j_h}\binom{k+1}{k_h} \\
    & = \sum_{\substack{j_v+k_v=s_v\\
    v=2,...,m}}^{i}\prod_{\substack{h=2\\
    h\neq i}}^{m}(-1)^{k_h}\binom{k+j_h}{j_h}\binom{k+1}{k_h}. \\
\end{split}
\end{equation}
This is because the sum over $(j_1,k_1)$ has only one term, where the values of these variables are zero, and thus can be factored out. This implies that
\begin{equation}
\begin{split}
    D_{n_j+1,k}(\mathcal{S}') & = \sum_{j_1+k_1=s_1+1}(-1)^{k_1}\binom{k+j_1}{j_1}\binom{k+1}{k_1}D_{n_j-s_1,k}(\mathcal{S}''). \\
\end{split}
\end{equation}
There are two distinct possibilities. If $s_1<n_j$, then $n_j-s_1 \geq 1$ and thus $D_{n_j-s_1,k}(\mathcal{S}'')$ fulfills the requirements for the inductive step. This means that $D_{n_j-s_1,k}(\mathcal{S}'')$ vanishes and so $D_{n_j+1,k}(\mathcal{S}')$ vanishes too. The other case, for which $s_1 = n_j$, does not fulfills the inductive step conditions (one needs $n_j>0$) and in fact turns out to be equal to one: $D_{0,k}(\mathcal{S}'') = 1$. This in turn implies that
\begin{equation}
\label{Non_trivial}
\begin{split}
    D_{n_j+1,k}(\mathcal{S}') & = \sum_{j_1+k_1=n_j+1}(-1)^{k_1}\binom{k+j_1}{j_1}\binom{k+1}{k_1} \\
    & = \sum_{k_1=0}^{n_j+1}(-1)^{k_1}\binom{k+n_j+1-k_1}{n_j+1-k_1}\binom{k+1}{k_1}. \\
\end{split}
\end{equation}
Although non-trivial, this expression is more manageable and its nullity can be proven once again by induction. This is done in \hyperref[AppendixA]{Appendix A}. With this, in either case, $D_{n_j+1,k}(\mathcal{S}')$ vanishes and the inductive step is fulfilled.

Having proven that $D_{n_j,k}(\mathcal{S})$ vanishes for any integer $n_j$ and any combination $\mathcal{S}$ with the given conditions, this consequently proves that $C_j = 0$ for $j = 0,...,n-1$ and, by means of Eq. (\ref{Expansion}), finalizes the proof that (\ref{Polynomial}) fulfills the conditions (\ref{property}) for the MTP. $\blacksquare$

\section{Discussion and conclusions}
A benefit of using a MTP instead of a STP is that one can obtain a greater accuracy for the interpolation of a function with the knowledge of less derivatives. That is, if one knows the first $k$ derivatives of a function at $m$ different points, the MTP possesses the same degree (and thus the same accuracy) with respect to the STP where a higher number of derivatives ($mk+m-1$) is known.

Although generally expensive to be computed, there can be parts of the expression (\ref{Polynomial}) which can be calculated without knowledge of the function to be interpolated, and so can be obtained beforehand and saved for optimization. Such terms include
\begin{equation}
\label{comp1}
    \bigg[\prod^{m}_{\substack{h=1\\
    h\neq g}}\frac{(x-a_h)}{(a_g-a_h)}\bigg]^{k+1}
\end{equation}
(which are powers of LP's factors), and
\begin{equation}
\label{comp2}
    \prod^{m}_{\substack{l=1\\
    l\neq i}}\frac{(k+j_l)!}{k!}(a_l-a_i)^{-j_l},
\end{equation}
where the second one is to be calculated for $0\leq j_l \leq k$.

As for the limiting behaviour, this is explored in Ref.~\cite{franssens1999} for general interpolating basis functions, where convergence to the function is found in appropriate analytic regions of the function (i) when the number of derivatives $k$ tends to infinity for fixed $m$, and (ii) when the spacing between the points becomes zero for fixed $k$.

Apart from discussing convergence of the series with $k$ going to infinity, the work on Ref.~\cite{fine1962} gives a general method for obtaining the MTP for analytic functions in terms of an expansion
\begin{equation}
    \sum_{t=0}^{k}[p(z)]^{t}p_{m-1,t}(z),
\end{equation}
where
\begin{equation}
    p(z) = \prod_{h=1}^{m}(z-a_h),
\end{equation}
and $p_{m-1,t}(z)$ are polynomials of degree $m-1$ whose coefficients can be calculated in terms of a contour integral that could be generally computed via the Cauchy residue theorem. Nonetheless, an explicit expression for such coefficients is not given.

An explicit expression for the MTP was in fact already introduced by J. L. López and N. M. Temme for analytical functions, first for two distinct points~\cite{2Point_N_Temme} and later for an arbitrary number of points (some of which could also be repeated)~\cite{MPoint_N_Temme}. In this work, the authors presented a Taylor series expansion at different points, considering the remainder of the finite polynomial and convergence radius. In the context of this paper their result applied for a finite polynomial and for different points is
\begin{equation}
\label{Q_km}
    P_{k,m}(z) = \sum_{n=0}^{k}q_{n,m}(z)\prod_{l=1}^{m}(z-a_l)^{n},
\end{equation}
where $q_{n,m}(z)$ are polynomials of degree $m-1$ given by
\begin{equation}
    q_{n,m}(z) \equiv \sum_{g=1}^{m}A_{n,g}\prod^{m}_{\substack{h=1\\
    h\neq g}}\frac{(z-a_h)}{(a_g-a_h)},
\end{equation}
and the constants $A_{n,g}$ can be given by the Cauchy integral
\begin{equation}
\label{A_ng_Cauchy}
    A_{n,g} = \frac{1}{2\pi i}\int_{\mathcal{C}}\frac{f(w)\,\mathrm{d}w}{(w-a_g)\prod_{h=1}^{m}(w-a_h)^n},
\end{equation}
where the contour of integration $\mathcal{C}$ is a simple closed loop which encircles all of the points of the set $\mathcal{A}$ of points in the counterclockwise direction and is contained in the analytical region $\Omega$ of the function. The aforementioned explicit expression for the MTP is obtained when one takes an alternate expression for these constants, given by the composite derivative
\begin{equation}
\label{A_ng2}
    A_{n,g} = \bigg[\frac{1}{n!}\frac{\mathrm{d}^{n}}{\mathrm{d}w^{n}}\frac{f(w)}{\prod^{m}_{\substack{h=1\\
    h\neq g}}(w-a_h)^{n}}\bigg]\bigg|_{w=a_g}+\sum_{\substack{l=1\\
    l\neq g}}^{m}\bigg[\frac{1}{(n-1)!}\frac{\mathrm{d}^{n-1}}{\mathrm{d}w^{n-1}}\frac{f(w)}{(w-a_g)\prod^{m}_{\substack{h=1\\
    h\neq l}}(w-a_h)^{n}}\bigg]\bigg|_{w=a_l},
\end{equation}
and which was also presented in this work. This formula holds for $n=1,2,3...$, while for the $n=0$ case one can directly obtain that $A_{0,g} = a_g$ as Eq. (\ref{A_ng_Cauchy}) results in the trivial Cauchy formula. Furthermore, the study also presents the multi-point generalization of the disk as the region of convergence for one point, given by
\begin{equation}
    O_m \equiv \{ z\in \Omega, \, \prod_{h=1}^{m}|z-a_h| < r\}, \qquad \qquad r \equiv \mathrm{Inf}_{w \in \mathbb{C}/\Omega}\bigg\{\prod_{h=1}^{m}|w-a_h|\bigg\}.
\end{equation}
It is also worth noting that a general Laurent series for multiple points is also analyzed, which is something that was not considered here.

Although the explicit expression of the polynomial (\ref{Q_km}) is not similar to the one presented here, the uniqueness of the solution must ensure that, under certain algebraic manipulations, Eqs. (\ref{Polynomial}) and (\ref{Q_km}) must be equal. The main difference between both results is that the expansion of both polynomials is performed in a different manner, which produces different expressions for the coefficients of each term. That is, the constants $A_{n,g}$ are expressed as n-th derivatives of quotients between the function $f(w)$ and powers of $(w-a_h)$, while the constants $F^{n,g}_{k,m}[\mathcal{A}]$ are expressed in terms of a multinomial sum.

The conclusion is thus that the expression presented here is novel in its form. The usefulness of either expression will depend on the specific purposes for which it is required. An example for where the polynomial (\ref{Polynomial}) is more applicable is in the context of interpolation problems where the numerical values of the function and its derivatives at specific points are known, but not its general form. Expanding the derivatives in the constant $A_{n,g}$ as given by Eq. (\ref{A_ng2}) would ultimately result in expressions similar to the factors $F^{n,g}_{k,m}[\mathcal{A}]$, but this can be computationally expensive to produce and the resulting terms would not be grouped so naturally (for example, additional algebraic manipulations would be needed to obtain the factors (\ref{comp2}) which are useful for optimizing the computation of the polynomial). Thus, the expression (\ref{Polynomial}) can be more appropriate for explicit computations.

\section*{Acknowledgements}
Thanks to Stefan Nellen for proofreading the manuscript.

\section*{Financial disclosure}
Acknowledgements to the project DGAPA-UNAM IN103319 for financial support.

\section*{Conflict of interest}
The author declares no potential conflict of interests.

Explicit Multi-point Taylor Polynomial


New bounds for Szemeredi's theorem, II: A new bound for $r_4(N)$

Abstract: Define $r_4(N)$ to be the largest cardinality of a set $A$ in $\{1,\dots,N\}$ which does not contain four elements in arithmetic progression. In 1998 Gowers proved that $r_4(N) \ll N(\log \log N)^{-c}$ for some absolute constant $c> 0$. In this paper (part II of a series) we improve this to $r_4(N) \ll N e^{-c\sqrt{\log \log N}}$. In part III of the series we will use a more elaborate argument to improve this to $r_4(N) \ll N(\log N)^{-c}$.

\section{Introduction}

\textsc{notational convention.} Throughout the paper the letters $c,C$ will denote absolute constants which could be specified explicitly if desired. These constants will generally satisfy $0 < c \ll 1 \ll C$. Different instances of the notation, even on the same line, will typically denote different constants. Occasionally we will want to fix a constant for the duration of an argument; such constants will be subscripted as $C_0,C_1$ and so on. Any implied constants in the $O$- or $\ll$ notations will depend only on any subscripted variables. Thus if we say that $f(N) = O_{\delta}(N)$ we mean that there is a constant $F(\delta)$ such that $f(N) \leq F(\alpha) N$ for all $N$. The absence of any subscripted variables should be taken to mean that the implied constant is absolute.

Let $N$ be a large positive integer, and let $k \geq 3$ be fixed. We define $r_k(N)$ to be the largest cardinality of a set $A \subseteq [N] = \{1,\dots,N\}$ which does not contain $k$ distinct elements in arithmetic progression.

Klaus Roth proved in 1953 \cite{roth} that
\[ r_3(N) \ll N(\log \log N)^{-1}.\]
In particular, $r_3(N) = o(N)$.
Since Szemer\'edi's 1969 proof that $r_4(N) = o(N)$ \cite{szemeredi-4}, and his later proof \cite{szemeredi} that $r_k(N) = o_k(N)$ for $k \geq 5$, it has been natural to ask for similarly effective bounds for these quantities. A first attempt in this direction was made by Roth in \cite{roth-4}, who provided a new proof that $r_4(N) = o(N)$. A major breakthrough was made by in 1998 by Gowers \cite{gowers-4-aps,gowers-long-aps}, who obtained the bound
\[ r_k(N) \ll N(\log \log N)^{-c_k}\] for each $k \geq 4$.

In the meantime, there has been progress on $r_3(N)$. Szemer\'edi (unpublished) obtained the bound
\begin{equation}\label{eq1.1} r_3(N) \ll Ne^{-c\sqrt{\log \log N}},\end{equation} and shortly thereafter
Heath-Brown \cite{heath} and Szemer\'edi \cite{szem-3ap} independently obtained the bound
\begin{equation}\label{eq1.2} r_3(N) \ll N(\log N)^{-c}.\end{equation}
More recently Bourgain \cite{Bou} found the best bound currently known, namely
\[ r_3(N) \ll N(\log \log N/\log N)^{1/2}.\]
Part I of this series of papers \cite{green-tao-szem} may be consulted for a more extensive discussion of the history of the problem. Our objective in this series is to bring our knowledge of $r_4$ more closely into line with the best known bounds for $r_3$. In \cite{green-tao-szem} this was achieved in the so-called \emph{finite field model}, in which $[N]$ is replaced by a vector space $\mathbb{F}_p^n$ over a finite field. In this paper we instead study subsets of $[N]$ itself, and obtain the analogue of Szemer\'edi's unpublished bound \eqref{eq1.1} for $r_4(N)$.

\begin{theorem}[Main theorem]\label{main}  For all large integers $N$ we have
\[ r_4(N) \ll N e^{- c \sqrt{\log \log N}}.\]
\end{theorem}

In part III of the series we will obtain the analogue of the superior bound \eqref{eq1.2}. The argument will, however, be substantially more technical.

Let us conclude this introduction by mentioning that the best known \emph{lower} bound for $r_4(N)$ is essentially the same as that for $r_3(N)$, namely Behrend's 1946 bound \cite{behrend}
\[ r_4(N) \geq r_3(N) \gg Ne^{-c\sqrt{\log N}}.\]
Somewhat better bounds of shape
\[ r_k(N) \gg Ne^{-(\log N)^{c_k}}\]
are known for much larger $k$: see  \cite{laba,rankin} for details.

We now briefly outline the proof of Theorem \ref{main}.  As with all previous papers obtaining quantitative bounds for $r_k(N)$, we use the \textit{density increment strategy} of Roth, a detailed discussion of which may be found in \cite{green-fin-field}.
The key is to obtain a dichotomy of the following form.

\begin{proposition}[Lack of progressions implies density increment]\label{densinc-general}  Let $N$ be a large integer, let $\delta \in (0,1)$, and suppose that $A \subseteq [N]$ has $|A| \geq \delta N$ and contains no progressions of length $4$.  Assume that we have the largeness condition $N \geq F(\delta)$ for some explicit function $F$.
Then there exists an arithmetic progression $P \subseteq [N]$ of length at least $f(N,\delta)$ on which we have the density increment
\[ \frac{|A \cap P|}{|P|} \geq \delta + \sigma(\delta).\]
Here $f(N,\delta) > 0$ is an explicit function which goes to $\infty$ as $N \to \infty$ for each fixed $\delta$, and $\sigma(\delta) > 0$ is an explicit positive quantity depending only on $\delta$.
\end{proposition}

Any proposition of this type will imply, by iteration, a nontrivial upper bound on $r_4(N)$, with the precise bound depending on the functions $F(\,)$, $f(\, , \,)$, and $\sigma(\,)$. For an actual calculation of a bound (on $r_4(\mathbb{F}_5^n)$) using this strategy, part I of this series may be consulted.

If one desires a good bound it is of particular interest to get $f(N,\delta)$ and $\sigma(\delta)$ as large as possible. The function $F(\delta)$ plays a much less significant r\^ole and, at least for the purposes of a motivating discussion, may be ignored.  Gowers' proof that $r_4(N) \ll N(\log \log N)^{-c}$ proceeds by establishing Proposition \ref{densinc-general} with
$f(N,\delta) \gg N^{c\delta^C}$ and $\sigma(\delta) \gg \delta^C$.
The main advance in our paper is to improve the density increment bound to $\sigma(\delta) \gg \delta$. This has the effect of reducing the number of iterations of Proposition \ref{densinc-general} that are required from $C\delta^{-C}$ to $C \log(1/\delta)$. Here is a more precise statement of what we shall prove.

\begin{proposition}[Lack of progressions implies density increment]\label{densinc-specific}  Let $\delta > 0$, and suppose that $N \geq e^{C \delta^{-C}}$. Let $A$ be a subset of $[N]$ with $|A| \geq \delta N$ such that
$A$ contains no progressions of length $4$.  Then there exists an arithmetic
progression $P$ in $[N]$ of length $|P| \gg N^{c \delta^C}$ such that we have the density increment
\[ \frac{|A \cap P|}{|P|} \geq (1+c) \delta.\]
\end{proposition}

Let us now quickly show how this implies Theorem \ref{main}.

\emph{Deduction of Theorem \ref{main} from Proposition \ref{densinc-specific}.}
Suppose that $A \subseteq [N]$ has size $\delta N$, and that it does not contain a 4-term progression. We perform an iteration. At the $i$th step of this iteration we will have a set $A_i \subseteq \{1,\dots,N_i\}$ with size $\delta_i N$.  This set will be a linearly rescaled version of a subset of $A$, and so it too does not contain a progression of length 4. Set $A_0 := A$, $N_0 := N$ and $\delta_0 := \delta$. Now Proposition \ref{densinc-specific} tells us that either
\begin{equation}\label{bottom-out}
N_i \leq e^{C\delta_i^{-C}}
\end{equation}
or else the iteration proceeds and it is possible to choose $N_{i+1}, \delta_{i+1}$ and $A_{i+1}$ such that\[ N_{i+1} \gg N_i^{c\delta_i^C}\]
and \[ \delta_{i+1} \geq (1 + c)\delta_i.\]
Now as long as the iteration continues we must have $\delta_i \leq 1$, and so after $K \leq C \log(1/\delta)$ iterations the condition \eqref{bottom-out} must be satisfied. At this point we have
\[ N_K \gg N^{(c\delta^C)^{C\log(1/\delta)}},\] and so we derive the inequality
\[ N^{(c\delta^C)^{C\log(1/\delta)}} \leq e^{C \delta^{-C}}.\]
After a small amount of rearrangement this leads to the claimed bound
\begin{equation}\tag*{\usebox{\proofbox}} r_4(N) \ll N e^{-c\sqrt{\log \log N}}.\end{equation}

It thus remains to establish Proposition \ref{densinc-specific}.  Our starting point is our earlier paper \cite{gt-inverseu3}, which built upon the original paper of Gowers \cite{gowers-4-aps} to provide an \emph{inverse $U^3$ theorem} which, among other things, already implies Gowers' bound $r_4(N) \ll N(\log \log N)^{-c}$.  This inverse theorem will be stated properly in later sections, but roughly speaking if $A \subset [N]$ had size $|A| \geq \delta N$
and had no progressions of length $4$, then $A$ would have significant correlation with a certain ``local quadratic phase function''. An example of such a function is $n \mapsto e^{2\pi i \alpha n^2}$, though this is not the most general example; the reader may wish to consult the surveys \cite{green-icm,tao-coates} for further discussion.

This correlation implies that $A$ has a significant density increment (comparable to $\delta^C$) on a ``quadratic Bohr set'', that is to say an approximate level set of a local quadratic phase function. Such a set has size $\sim \delta^C N$. The next step in \cite{gowers-4-aps} is then that of \emph{linearisation}, in which the Bohr set is partitioned into arithmetic progressions. By the pigeonhole principle $A$ also has a density increment of $\sim \delta^C$ on one of these progressions. It turns out that the linearisation can be achieved with progressions of size $\gg N^{c\delta^C}$ which, as mentioned earlier, is sufficient to give Gowers' bound. We remark that a similar linearisation step already appears in the earlier work of Roth \cite{roth} (cf. \cite[Lemma 2.3]{gowers-long-aps}).

The main cost in this scheme lies in the linearisation step, which forces one to pass from an object of size $N$ to an object of size only $N^{c\delta^C}$.
To improve upon this scheme we borrow an idea of Heath-Brown and Szemer\'edi \cite{heath,szem-3ap} from the $k=3$ case. Instead of finding a quadratic phase function which correlates with $A$ and then linearizing, one adopts a more patient stance and first collects \emph{several} quadratic phase functions. In this way a more substantial density increment of $c\delta$ can be obtained. Only after this is done do we linearise. This procedure of linearizing several quadratic phase functions at once turns out not to be as costly as one might think, and in any case it need only be done $O(\log (1/\delta))$ times due to the size of the density increment.

In part III of the series we will show that it is possible to be more efficient still, by extracting additional gains either from the density increment or from the length of the progression on which the increment is obtained. This was carried out in the finite field setting in \cite{green-tao-szem}.

We have mentioned, albeit briefly, the so-called finite field model: the survey \cite{green-fin-field} may be consulted for more information. The advantage of working in $\mathbb{F}_p^n$ as opposed to the cyclic group $\mathbb{Z}/N\mathbb{Z}$ (which serves as a model for $[N]$) is the availability of \emph{subspaces}. In $\mathbb{Z}/N\mathbb{Z}$, and in other abelian groups $G$, one must make do with the notion of \emph{Bohr sets}, which may be thought of as approximate subspaces. There are various technical issues involved in dealing with these, as we shall see later on.

\begin{remark} It is quite likely that the methods here combine with those in \cite{green-tao-szem} extend to general finite abelian groups $G$; thus if $r_4(G)$ denotes the largest cardinality $|A|$ of a set $A \subset G$ without any arithmetic progressions of length $4$, a slight elaboration of the arguments here should establish $r_4(G) \ll |G| e^{- c \sqrt{\log \log |G|} }$ for all large $|G|$. We will however not pursue this matter here.
\end{remark}

\section{General notation}\label{notation-sec}

Let $A$ be a finite non-empty set and let $f: A \to \mathbb{C}$ be a function. It is convenient, so as to avoid having to contend with normalising factors, to use the expectation notation
\[\mathbb{E}_A(f) = \mathbb{E}_{x \in A} f(x) := \frac{1}{|A|} \sum_{x \in A} f(x).\]
More complex expressions such as $\mathbb{E}_{x \in A, y \in B} f(x,y)$ are similarly defined.  We also define the $L^p$ norms
$$ \|f\|_{L^p(A)} := (\mathbb{E}_A|f|^p)^{1/p}$$
for $1 \leq p < \infty$, with the usual convention $\|f\|_{L^\infty(A)} := \sup_{x \in A} |f(x)|$.
We say that $f$ is \emph{$1$-bounded} if $\|f\|_{L^\infty(A)} \leq 1$.

If $A,B$ are finite sets with $B$ non-empty, we write $\mathbb{P}_B(A) := \frac{|A \cap B|}{|B|}$ for the density of $A$ in $B$. If $A$ lies in some ambient space $X$ (for example a group) we use $1_A: X \to \mathbb{R}$ to denote the
indicator function of $A$, that is to say $1_A(x) = 1$ when $x \in A$
and $1_A(x) = 0$ otherwise. We also write $1_{x \in A}$ for $1_A(x)$.  Thus for instance $\mathbb{P}_B(A) = \mathbb{E}_B(1_A)$ for all
non-empty $B \subseteq X$.

\section{The form $\Lambda$ and the $U^3(\mathbb{Z}/p\mathbb{Z})$ norm}\label{lambda-sec}

We now begin the proof of Proposition \ref{densinc-specific}.  It will be convenient to work in a cyclic group $\mathbb{Z}/p\mathbb{Z}$ of
large prime order $p$ rather than on the interval $[N]$.  On this cyclic group $\mathbb{Z}/p\mathbb{Z}$, we introduce the quadrilinear form $\Lambda(f_0,f_1,f_2,f_3)$, defined for four functions
$f_j: \mathbb{Z}/p\mathbb{Z} \to \mathbb{C}$ by
\[ \Lambda(f_0,f_1,f_2,f_3) := \mathbb{E}_{x,h \in \mathbb{Z}/p\mathbb{Z}} f_0(x) f_1(x+h) f_2(x+2h) f_3(x+3h).\]
This form is clearly pertinent to the task of counting progressions of length $4$, and has appeared in many previous papers on this subject.
One can quickly deduce Proposition \ref{densinc-specific}, and hence Theorem \ref{main}, from the following claim.

\begin{theorem}[Anomalous number of AP4s implies density increment]\label{dens-inc}
Let $p$ be a large prime, let $N$ be an integer between $p/8$ and $p/4$, and let $f: \mathbb{Z}/p\mathbb{Z} \to \mathbb{R}$ be a $1$-bounded non-negative
function which vanishes outside of $[N]$.  Set $\delta := \mathbb{E}_{[N]}(f)$.  Suppose that
\begin{equation}\label{plarge}
p \gg \exp( C \delta^{-C} )
\end{equation}
for some suitably large
absolute constant $C$, and suppose that
\begin{equation}\label{law}
|\Lambda(f,f,f,f) - \Lambda(\delta 1_{[N]}, \delta 1_{[N]}, \delta 1_{[N]}, \delta 1_{[N]})| \gg \delta^4.
\end{equation}
Then we can find an arithmetic progression $P$ in $[N]$ obeying the length bound
\begin{equation}\label{qinc}
|P| \gg p^{c \delta^C}
\end{equation}
and the density increment bound
\begin{equation}\label{pinc}
 \mathbb{E}_P(f) \geq (1+c) \delta
\end{equation}
for some $c, C > 0$.
\end{theorem}

\begin{remark}
Strictly speaking, there could be two different notions of an arithmetic progression in $[N]$, one arising from its embedding into the integers $\mathbb{Z}$, and the other arising from its embedding into the cyclic group $\mathbb{Z}/p\mathbb{Z}$.  However, because $N < p/4$, it is easy to see that the two concepts are equivalent; the interval $[N]$ is too short to contain a progression that somehow ``wraps around'' $p$.  (To use some jargon, the two representations of $[N]$ are \emph{Freiman isomorphic} of order $2$, which is sufficient to preserve the concept of an arithmetic progression; see for instance \cite{taovu-book}.)
\end{remark}

\emph{Proof that Proposition \ref{densinc-specific} implies Theorem \ref{dens-inc}.}  By increasing $\delta$ if necessary we may
assume that $|A| = \delta N$.  Choose a prime $p$ between $4N$ and $8N$ (this is of course possible by Bertrand's Postulate)
and take $f := 1_A$, thought of as a function on $\mathbb{Z}/p\mathbb{Z}$.  Since $A$ has no progressions of length $4$ we easily see that
\[ \Lambda(f,f,f,f)= O( 1/p ),\]
whilst the fact that there are $\frac{1}{6}N^2 (1 + o(1))$ four-term progressions in $[N]$ implies that
\[ \Lambda(\delta 1_{[N]}, \delta 1_{[N]}, \delta 1_{[N]}, \delta 1_{[N]}) \gg \delta^4.\]
Since we are taking $p$ to be large, we conclude \eqref{law}.  Applying Theorem \ref{dens-inc}, we can find a progression $P$ in $\mathbb{Z}/p\mathbb{Z}$
obeying \eqref{qinc} and \eqref{pinc}, and this suffices for our needs.  \hfill{\usebox{\proofbox}}

It remains to prove Theorem \ref{dens-inc}.  For the rest of the paper we fix $p$ to be a large prime.
To be able to exploit the hypothesis \eqref{law}, we will need to show that $\Lambda$ is controlled by either of two norms (when restricted
to $1$-bounded functions).  The first is the $L^1$ norm.

\begin{lemma}[$L^1$ controls $\Lambda$]\label{l1-lambda4} Let $f, g: \mathbb{Z}/p\mathbb{Z} \to \mathbb{C}$ be uniformly bounded by some $\alpha > 0$.
Then we have
\[ |\Lambda(f,f,f,f) - \Lambda(g,g,g,g)| \leq 4 \alpha^3 \|f-g\|_{L^1(\mathbb{Z}/p\mathbb{Z})}.\]
\end{lemma}
\noindent\textit{Proof. }
Since $\Lambda$ is quadrilinear we have
\begin{align}\nonumber \Lambda(f,f,f,& f) - \Lambda(g,g,g,g) \\&= \Lambda(f-g,f,f,f) + \Lambda(g,f-g,f,f) + \Lambda(g,g,f-g,f) + \Lambda(g,g,g,f-g).\label{telescope}\end{align}
The result now follows on applying the triangle inequality and the easily checked bound
\begin{equation}\label{fairly-trivial} |\Lambda(f_1,f_2,f_3,f_4)| \leq \Vert f_j \Vert_1 \sup_{i = 1,\dots,4} \Vert f_i \Vert_{\infty}^3,\end{equation}
valid for $j = 1,\dots,4$.\hfill{\usebox{\proofbox}}

The second norm that controls $\Lambda$ is the \emph{Gowers $U^3$-norm} $\|f\|_{U^3(\mathbb{Z}/p\mathbb{Z})}$ of a function $f: \mathbb{Z}/p\mathbb{Z} \to \mathbb{C}$, defined as
\begin{eqnarray*} \Vert f \Vert_{U^3(\mathbb{Z}/p\mathbb{Z})}^8 & := & \mathbb{E}_{x,h_1,h_2,h_3 \in \mathbb{Z}/p\mathbb{Z}}(f(x)\overline{f(x+h_1)f(x+h_2)f(x+h_3)}f(x+h_1+h_2) \times \\
& & \qquad\qquad\qquad\qquad \times f(x+h_2+h_3)f(x+h_1+h_3) \overline{f(x+h_1+h_2+h_3)}).\end{eqnarray*}
This norm was introduced in \cite{gowers-4-aps,gowers-long-aps} and
studied further in such papers as \cite{green-tao-primes,gt-inverseu3,green-tao-szem,taovu-book}.  As shown in \cite{gowers-long-aps} it is indeed a norm on $\mathbb{Z}/p\mathbb{Z}$, but we will not need to know this here.  In fact we only require two facts about the $U^3$-norm. One of these facts is an \emph{inverse theorem}, which will be the subject of the next section.
The other is that the $U^3$-norm controls $\Lambda$.

\begin{lemma}[$U^3$ controls $\Lambda$]\label{u3-lambda4}
Let $f, g: \mathbb{Z}/p\mathbb{Z} \to \mathbb{C}$ be $1$-bounded functions on an affine space $\mathbb{Z}/p\mathbb{Z}$.  Then we have
$$ |\Lambda(f,f,f,f) - \Lambda(g,g,g,g)| \leq 4 \|f-g\|_{U^3(\mathbb{Z}/p\mathbb{Z})}.$$
\end{lemma}
\noindent\textit{Proof. } We employ the same telescoping identity \eqref{telescope} that we use to prove Lemma \ref{l1-lambda4}. In place of the fairly trivial bound \eqref{fairly-trivial} we instead apply the \emph{Generalized von Neumann theorem}, which in this setting states that
\[ |\Lambda(f_1,f_2,f_3,f_4)| \leq \Vert f_j \Vert_{U^3(\mathbb{Z}/p\mathbb{Z})}\]
for $j = 1,\dots,4$. This result is proved using three applications of the Cauchy-Schwarz inequality: the details are given very explicitly in \cite[Proposition 1.11]{green-montreal}.\hfill{\usebox{\proofbox}}

In \cite{gowers-4-aps,gowers-long-aps} one applied Lemma \ref{u3-lambda4} directly to \eqref{law} in order to obtain the lower bound
$\| f - \delta 1_{[N]} \|_{U^3(\mathbb{Z}/p\mathbb{Z})} \gg \delta^3$.  This ultimately led to a density increment of $\gg \delta^C$ for $f$ on some progression. The resulting iteration scheme thus proceeds for $\gg \delta^{-C}$ steps, which is too long for our purposes. Our approach is to develop a so-called \emph{Koopman-von Neumann structure theorem}, which introduces an intermediate approximant $\mathbb{E}(f|\mathcal{B})$ between $f$ and $\delta 1_{[N]}$.

\section{The inverse $U^3(\mathbb{Z}/p\mathbb{Z})$ theorem}\label{inverse-sec}

We now come to the second fact concerning the $U^3(\mathbb{Z}/p\mathbb{Z})$-norm that we shall need, namely the \emph{inverse $U^3$-theorem}. This is one of the main results of \cite{gt-inverseu3}.  There are three (equivalent) formulations of this inverse theorem: one involving locally quadratic phase functions, one involving generalized quadratic phases, and one involving $2$-step nilsequences.  Our argument would work with the first two of these but not the third (cf. \cite[Theorem 12.7]{gt-inverseu3}), which has rather weaker bounds. We use the first formulation involving locally quadratic phases. This is in a sense the most basic form of the inverse theorem for $U^3(\mathbb{Z}/p\mathbb{Z})$, since in \cite{gt-inverseu3} the other variants are all derived from it. To describe the result we need some notation.

\begin{definition}[Bohr sets]\label{bohr-set}
Let $S \subseteq \mathbb{Z}/p\mathbb{Z}$, and let $\rho \in (0,1)$ be a parameter. We define the \emph{\textup{(}centred\textup{)} Bohr set} $B(S,\rho) \subseteq \mathbb{Z}/p\mathbb{Z}$ to be the set
\[ B(S,\rho) := \{ x \in \mathbb{Z}/p\mathbb{Z} : \|\xi x / p\|_{\mathbb{R}/\mathbb{Z}} < \rho \},\]
where $\|x\|_{\mathbb{R}/\mathbb{Z}}$ denotes the distance from $x$ to the nearest integer.  More generally, if $\alpha = (\alpha_\xi)_{\xi \in S}$ is any element in the $|S|$-dimensional torus $(\mathbb{R}/\mathbb{Z})^S$, then we write $B_\alpha(S,\rho)$ for the \emph{uncentred Bohr set}
\[ B_\alpha(S,\rho) := \{ x \in \mathbb{Z}/p\mathbb{Z} : \|\xi x / p - \alpha_\xi \|_{\mathbb{R}/\mathbb{Z}} < \rho \}.\]
We refer to $|S|$ as the \emph{rank} of the Bohr set, and $\rho$ as the \emph{radius}.
\end{definition}

\begin{example} The arithmetic progression $[N]$ is an uncentred Bohr set of rank $1$, with $S = \{1\}$, $\alpha_1 = (N+1)/2p$ and $\rho = N/2p$.
More generally, any arithmetic progression is an uncentred Bohr set of rank $1$, and conversely.  The intersection of $d$ arithmetic progressions of equal length will be an uncentred Bohr set of rank $d$.  (In fact, in a cyclic group of prime order, this essentially describes all the possible uncentred Bohr sets.)
\end{example}

The inverse $U^3$-theorem will only require the centred Bohr sets, but we will need the uncentred Bohr sets in the next section, when we convert the inverse theorem into a Koopman-von Neumann type structure theorem.

Dealing with Bohr sets is slightly technical. One reason for this is that $|B(S,\rho)|$ is not guaranteed to depend particularly smoothly on $\rho$. As discovered by Bourgain \cite{Bou} (see also \cite[Chapter 8]{gt-inverseu3}), such a property \emph{can} be guaranteed for a large supply of $\rho$. To discuss this issue, the following definition is pertinent.

\begin{definition}[Regular Bohr sets]\label{regular-def} Let $S \subseteq \mathbb{Z}/p\mathbb{Z}$ be a set with size $d = |S|$, and suppose that $0 < \rho < 1/2$. A Bohr set $B(S,\rho)$ is said to be \emph{regular} if one has
\[ (1-100d |\kappa|) |B(S,\rho)| \leq |B(S, (1+\kappa)\rho)| \leq (1+100d |\kappa|) |B(S,\rho)|\]
whenever $|\kappa| \leq 1/100d$.
\end{definition}

The \emph{raison d'\^etre} for this definition is a result of Bourgain \cite{Bou} (see also \cite[Lemma 8.2]{gt-inverseu3}) which states that for any $S$ and any $\varepsilon$ there is at least one regular value of $\rho$ in the interval $[\varepsilon,2\varepsilon]$. This will not concern us here though it was important for the proofs in \cite{gt-inverseu3}.

We move swiftly on to some other concepts which are useful in the discussion of the inverse theorem for the $U^3(\mathbb{Z}/p\mathbb{Z})$-norm.

\begin{definition}[Linear phase functions]  We say that a function $\phi: \mathbb{Z}/p\mathbb{Z} \to \mathbb{R}/\mathbb{Z}$ is a \emph{globally linear phase function} if we have
\[ \phi(x+h_1+h_2) - \phi(x+h_1) - \phi(x+h_2) + \phi(x) = 0\]
for all $x,h_1,h_2 \in \mathbb{Z}/p\mathbb{Z}$.
\end{definition}

\begin{example}
Because $p$ is prime, it is easy to see that a function $\phi: \mathbb{Z}/p\mathbb{Z} \to \mathbb{R}/\mathbb{Z}$ is globally linear if and only if it takes the form $\phi(x) = \xi x/p + \alpha$ for some $\xi \in \mathbb{Z}/p\mathbb{Z}$ and $\alpha \in \mathbb{R}/\mathbb{Z}$.
\end{example}

\begin{definition}[Quadratic phase functions]  Let $B \subset \mathbb{Z}/p\mathbb{Z}$.  We say that a function $\phi: B \to \mathbb{R}/\mathbb{Z}$ is a \emph{locally quadratic phase function} on $B$ if we have
\begin{equation}\label{quaddef}
\begin{split}
 \phi(x+h_1+h_2+h_3) &- \phi(x+h_1+h_2) - \phi(x+h_2+h_3) - \phi(x+h_1+h_3) \\
 &+ \phi(x+h_1) + \phi(x+h_2) + \phi(x+h_3) - \phi(x) = 0
\end{split}
\end{equation}
whenever $x$, $x+h_1$, $x+h_2$, $x+h_3$, $x+h_1+h_2$, $x+h_1+h_3$, $x+h_2+h_3$, and $x + h_1 + h_2 + h_3$ all lie in $B$.
\end{definition}

\begin{example}  Every globally linear phase function is locally quadratic.
If $\alpha,\beta,\gamma$ are real numbers, and $N$ is an integer between $p/8$ and $p/4$,
then the function $\phi(n) = \alpha n^2 + \beta n + \gamma \md{1}$ is a locally quadratic phase function on $[N]$.
\end{example}

\begin{remark}
There are also notions of locally linear phase functions, and globally quadratic ones, but we will not need them here.
\end{remark}

We are now ready to state the inverse theorem for the $U^3(\mathbb{Z}/p\mathbb{Z})$-norm in the form that we shall need it. We write $e(x) := e^{2\pi i x}$ as usual.

\begin{theorem}[Inverse $U^3(\mathbb{Z}/p\mathbb{Z})$ theorem] \label{invert-u3}
Let $f: \mathbb{Z}/p\mathbb{Z} \to \mathbb{C}$ be a $1$-bounded function
such that $\|f\|_{U^3(\mathbb{Z}/p\mathbb{Z})} \geq \eta$ for some $\eta \in (0,1)$.  Then there exists a regular Bohr set $B := B(S,\rho)$
with $|S| \ll \eta^{-C}$ and $\rho \gg \eta^C$, and a locally quadratic phase function $\phi_y: y + B \to \mathbb{R}/\mathbb{Z}$
on $y + B$ for every $y \in \mathbb{Z}/p\mathbb{Z}$, such that
\begin{equation}\label{efphi-general}
\mathbb{E}_{y \in \mathbb{Z}/p\mathbb{Z}} |\mathbb{E}_{t \in y+B} f(t) e(-\phi_y(t))| \gg \eta^C.
\end{equation}
\end{theorem}

This is \cite[Theorem 2.7]{gt-inverseu3}, where in fact the explicit value of $C = 2^{24}$ was attained.

\begin{remark} It is unfortunately necessary to deal with locally quadratic phase functions rather than the more intuitively natural
globally quadratic phase functions; see \cite{gowers-4-aps,green-icm,gt-inverseu3} for further discussion of issues of this type, or
the paper \cite{furstenberg-von-neumann} for a rather different perspective on the same phenomenon.
\end{remark}

\section{Linear and quadratic factors, and a quadratic Koopman-von Neumann theorem}\label{kvn-sec}

As in \cite{green-tao-szem}, we now use an ``energy increment argument'' to convert our inverse theorem to a quadratic structure theorem of Koopman-von Neumann type,
inspired by some ideas from ergodic theory. Part I of the series \cite{green-tao-szem} or the lecture notes \cite{green-montreal} may be consulted for further discussion, and \cite{tao:icm} gives a more general discussion of structure theorems and inverse theorems.  We first need some more notation.

\begin{definition}[Factors]  Let $W$ be any non-empty finite set.
Define a \emph{factor} (or \emph{$\sigma$-algebra})
in $W$ to be a collection $\mathcal{B}$ of subsets of $W$ which are closed under union, intersection, and complement, and which contains $\emptyset$ and $W$. Define an \emph{atom} of $\mathcal{B}$ to be a minimal non-empty subset of $W$; these partition $W$, and indeed in this finitary setting a factor may be thought of simply as a partition of $W$.  If $\mathcal{B}$, $\mathcal{B}'$ are factors in $W$ with $\mathcal{B} \subseteq \mathcal{B}'$ we say that $\mathcal{B}'$ \emph{extends} $\mathcal{B}$.  More generally, if $\mathcal{B}, \mathcal{B}'$ are factors in $W$ we let $\mathcal{B} \vee \mathcal{B}'$ be the smallest common extension, so that the atoms of $\mathcal{B} \vee \mathcal{B}'$ are the intersections of atoms of $\mathcal{B}$ and atoms of $\mathcal{B}'$.  If $\mathcal{B}$ is a factor in $W$ and $W'$ is a subset of $W$, we define the \emph{restriction}
$\mathcal{B}|_{W'}$ of $\mathcal{B}$ to $W'$ to be the factor of $W'$ formed by intersecting all the sets in $\mathcal{B}$ with $W'$.
If $f: W \to \mathbb{C}$, we let $\mathbb{E}(f|\mathcal{B}): W \to \mathbb{C}$ denote the conditional expectation
$$ \mathbb{E}(f|\mathcal{B})(x) := \mathbb{E}(f|\mathcal{B}(x)) \hbox{ for all } x \in W,$$
where $\mathcal{B}(x)$ is the unique atom in $\mathcal{B}$ that contains $x$.  Equivalently, $\mathbb{E}(f|\mathcal{B})$ is the orthogonal projection to the space $\mathcal{B}$-measurable functions in the Hilbert space
$L^2(\mathbb{Z}/p\mathbb{Z})$.
\end{definition}

We will focus our attention on very structured factors, namely linear and quadratic factors, which are generated from globally linear
and locally quadratic phase functions respectively.  The notation here is inspired by the finite field analogues in
\cite{green-tao-szem} but with one new parameter, a ``resolution'' $K$, which is needed as a substitute for the small torsion that one
enjoys in the finite field geometry setting. We first need to describe how to convert a phase function into a factor.

\begin{definition}\label{def-52} Call a phase function \emph{irrational} if it only takes irrational values.  If $\phi: W \to \mathbb{R}/\mathbb{Z}$ is an irrational
phase function on a finite nonempty set $W$ and $K \geq 1$ is an integer, we define $\mathcal{B}_{\phi,K}$ to be the factor in $W$ whose atoms are the sets
$\{ x \in \mathbb{Z}/p\mathbb{Z}: \| \phi(x) - j/K \|_{\mathbb{R}/\mathbb{Z}} < 1/2K \}$ for $j=0,1,\ldots,K-1$.
\end{definition}

\begin{remark} The assumption of irrationality is a minor technicality, used in order to avoid having to deal with the borderline case when $\| \phi(x) - j/K \|_{\mathbb{R}/\mathbb{Z}}$ is exactly equal to $1/2K$; in practice we shall be able to use perturbation arguments to work purely with irrational phase functions.
\end{remark}

\begin{definition}[Linear factors]  A \emph{linear factor of complexity at most $d$ and resolution $K$} is any factor $\mathcal{B}$ in $\mathbb{Z}/p\mathbb{Z}$
of the form $\mathcal{B} = \mathcal{B}_{\phi_1,K} \vee \ldots \vee \mathcal{B}_{\phi_{d'},K}$, where $0 \leq d' \leq d$ and $\phi_1, \ldots, \phi_{d'}: \mathbb{Z}/p\mathbb{Z} \to \mathbb{R}/\mathbb{Z}$ are irrational globally linear phase functions.
\end{definition}

\begin{remark} From the definitions we see that if $\mathcal{B}$ is a linear factor of complexity at most $d$ and resolution $K$, then $\mathcal{B}$ has at most $K^d$ atoms, each of which is an uncentred Bohr set of rank at most $d$ and radius $1/2K$.  Also, if $\mathcal{B}'$ is another linear factor of complexity at most $d'$ and resolution $K$, then clearly
$\mathcal{B} \vee \mathcal{B}'$ is a linear factor of complexity at most $d+d'$ and resolution $K$.
\end{remark}

\begin{definition}[Quadratic factors]  Let $B$ be an uncentred Bohr set.  A \emph{pure quadratic factor of complexity at most $d$ and resolution $K$}
in $B$ is any factor $\mathcal{B}$ in $B$ of the form $\mathcal{B} = \mathcal{B}_{\phi_1,K} \vee \ldots \vee \mathcal{B}_{\phi_{d'},K}$, where $0 \leq d' \leq d$ and $\phi_1, \ldots, \phi_d: B \to \mathbb{R}/\mathbb{Z}$ are irrational locally quadratic phase functions on $B$.  A \emph{quadratic factor of complexity at most $(d_1,d_2)$ and resolution $K$} is any pair $(\mathcal{B}_1,\mathcal{B}_2)$ of factors in $W$, where $\mathcal{B}_1$ is a linear factor of complexity at most $d_1$ and resolution at most $K$, and $\mathcal{B}_2$ is an extension of $\mathcal{B}_1$, whose restriction to any atom $B$ of $\mathcal{B}_1$ is a pure quadratic factor on $B$ of complexity at most $d_2$ and resolution at most $K$.
We say that one quadratic factor $(\mathcal{B}'_1,\mathcal{B}'_2)$ is a \emph{quadratic extension} of another $(\mathcal{B}_1,\mathcal{B}_2)$ if
$\mathcal{B}_1 \subseteq \mathcal{B}'_1$ and $\mathcal{B}_2 \subseteq \mathcal{B}'_2$.
\end{definition}

\begin{remark}
Observe that if $(\mathcal{B}_1,\mathcal{B}_2)$ and $(\mathcal{B}'_1,\mathcal{B}'_2)$ are quadratic factors of resolution $K$ and complexity at most $(d_1,d_2)$ and $(d'_1,d'_2)$ respectively,
then their common extension $(\mathcal{B}_1 \vee \mathcal{B}'_1, \mathcal{B}_2 \vee \mathcal{B}'_2)$ is a quadratic factor of complexity at most $(d_1+d'_1,d_2+d'_2)$; this is ultimately
because the restriction of a locally quadratic phase function to a smaller set remains locally quadratic.\end{remark}
Our next task is to rephrase the inverse theorem, Theorem \ref{invert-u3}, in terms of quadratic factors. At heart this is really nothing more than an averaging argument, though due to ``edge effects'' it is somewhat tedious to write down rigorously.

\begin{theorem}[Inverse theorem for $U^3(\mathbb{Z}/p\mathbb{Z})$, again]\label{invert-u3-sigma}
Let $f: \mathbb{Z}/p\mathbb{Z} \to \mathbb{C}$ be a $1$-bounded function such that $\|f\|_{U^3(\mathbb{Z}/p\mathbb{Z})} \geq \eta$ for some $\eta \in (0,1)$.  Suppose also
that $K$ is an integer such that $K \geq C \eta^{-C}$ for some sufficiently large constant $C > 0$.  Then there exists a quadratic factor $(\mathcal{B}_1,\mathcal{B}_2)$
in $\mathbb{Z}/p\mathbb{Z}$ of complexity at most $(O( \eta^{-C} ), 1)$ and resolution $K$ such that
\begin{equation}\label{fib}
\| \mathbb{E}(f|\mathcal{B}_2) \|_{L^1(\mathbb{Z}/p\mathbb{Z})} \gg \eta^C.
\end{equation}
\end{theorem}

\begin{proof}  Let $S, \rho$ be as in Theorem \ref{invert-u3}.
Let $\alpha = (\alpha_\xi)_{\xi \in S}$ be a point on the torus $(\mathbb{R}/\mathbb{Z})^S$ with irrational coefficients (one could chose it randomly, if desired).
We then define $\mathcal{B}_1$ to be the $\sigma$-algebra whose atoms are of the form
$$ \{ x \in \mathbb{Z}/p\mathbb{Z}: \| x \xi/p - \alpha_\xi - j_\xi / K \|_{\mathbb{R}/\mathbb{Z}} < 1/2K \hbox{ for all } \xi \in S \}$$
where for each $\xi \in S$, $j_\xi$ is an integer between $0$ and $K-1$.  One easily verifies that $\mathcal{B}_1$ is an irrational linear factor of complexity $|S|$ and resolution $K$, defined by linear phases $\phi_{\xi}(x) := x\xi/p - \alpha_{\xi}$, $\xi \in S$.  For each $x \in\mathbb{Z}/p\mathbb{Z}$, let $F(x)$ be the quantity
\[ F(x) := \sup_{\phi \in \Phi(x)} |\mathbb{E}_{t \in \mathcal{B}_1(x)}( f(t) e(-\phi(t)) )|,\]
where $\mathcal{B}_1(x)$ is the atom of $\mathcal{B}_1$ that contains $x$ and $\Phi(x)$ is the collection of all locally quadratic phase functions $\phi: \mathcal{B}_1(x) \to \mathbb{R}/\mathbb{Z}$. Thus $F$ measures the maximum correlation of $f$ with a quadratic phase on the atom $\mathcal{B}_1(x)$.  We claim that it suffices to show that
\begin{equation}\label{fl1}
\| F \|_{L^1(\mathbb{Z}/p\mathbb{Z})} \gg \eta^C.
\end{equation}
Suppose that this has been established. Written out in full, it becomes the statement that
\[ \mathbb{E}_{x \in \mathbb{Z}/p\mathbb{Z}} \big| \mathbb{E}_{t \in \mathcal{B}_1(x)} f(t) e(-\phi_{\mathcal{B}_1(x)}(t))  \big| \gg \eta^C\] for an appropriate choice of $\phi_{\mathcal{B}_1(x)} \in \Phi(x)$.
Modulating each phase by a complex number $e(\theta_x)$ we may move the modulus signs to the outside, obtaining
\[ \big|\mathbb{E}_{x \in \mathbb{Z}/p\mathbb{Z}} \mathbb{E}_{t \in \mathcal{B}_1(x)} f(t) e(-\phi_{\mathcal{B}_1(x)}(t))  \big|\gg \eta^C.\]
Since the two averaging operations are equivalent to the single averaging $\mathbb{E}_{x \in \mathbb{Z}/p\mathbb{Z}}$ this becomes
\begin{equation}\label{use-soon-1} |\mathbb{E}_{x \in\mathbb{Z}/p\mathbb{Z}} f(x) e(-\phi_{\mathcal{B}_1(x)}(x)) | \gg \eta^C.\end{equation}
By perturbing each of the $\phi_{B_1}$ infinitesimally we may assume that the $\phi_{B_1}$ are all irrational.  If we then let $\mathcal{B}_2$ be the extension of $\mathcal{B}_1$ whose restriction to each atom $B_1$ of $\mathcal{B}_1$ is given by $\mathcal{B}_{\phi_{B_1},K}$, then $(\mathcal{B}_1,\mathcal{B}_2)$ is a quadratic factor of complexity
at most $(|S|, 1)$ and resolution $K$, and we have
\[ e(-\phi_{\mathcal{B}_1(x)}(x)) = \mathbb{E}( e(-\phi_{\mathcal{B}_1(x)}) | \mathcal{B}_2 )(x)+ O(1/K)\]
for all $x$. It is important to note here that $\phi_{\mathcal{B}_1(x)}$ depends only on the atom $\mathcal{B}_1(x)$ and not otherwise on $x$ itself.

It follows from this and \eqref{use-soon-1} that if  $K \geq C \eta^{-C}$ for sufficiently large $C$ then
\[ \big|\langle f, \mathbb{E}( e(\phi_{\mathcal{B}_1(x)})|\mathcal{B}_2)\rangle\big| \gg \eta^C,\]
where we have written $\langle g_1,g_2 \rangle := \mathbb{E}_{x \in \mathbb{Z}/p\mathbb{Z}} g_1(x) \overline{g_2(x)}$.
The conditional expectation operator $g \mapsto \mathbb{E}(g | \mathcal{B}_2)$ is self-adjoint with respect to this inner product, and hence this implies that
\[ \big| \langle \mathbb{E}(f | \mathcal{B}_2), e(\phi_{\mathcal{B}_1(x)}) \rangle \big| \gg \eta^C.\]
The desired bound \eqref{fib} is now a consequence of the triangle inequality in the form $|\langle g_1,g_2\rangle| \leq \Vert g_1 \Vert_1 \Vert g_2 \Vert_{\infty}$.

It remains, then, to establish \eqref{fl1}. It is now time to exploit the estimate \eqref{efphi-general}, which we urge the reader to recall now.  For any fixed $y \in \mathbb{Z}/p\mathbb{Z}$, we write $\Omega_y$ for the union of those atoms of $\mathcal{B}_1$ which only partially intersect $y + B$ (thus they are neither contained in $y + B(S,\rho)$ nor outside of it).  We have
\begin{align}\nonumber
|\mathbb{E}_{t \in y+B} f(t) e(-\phi_y(t))| &\leq \sum_{B_1: B_1 \subset y+B} \frac{|B_1|}{|y+B|} |\mathbb{E}_{t \in B_1} f(t) e(-\phi_y(t))| + \mathbb{P}_{y+B}(\Omega_y) \\ \nonumber
&\leq \sum_{B_1: B_1 \subset y+B} \frac{|B_1|}{|y+B|} \mathbb{E}_{B_1} F + \mathbb{P}_{y+B}(\Omega_y) \\
&\leq \mathbb{E}_{y+B} F + \mathbb{P}_{y+B}(\Omega_y)\label{previous-bound}
\end{align}
(note that $F$ is constant on each atom $B_1$).
On the other hand, since $B = B(S,\rho)$, one easily verifies that
\[ \Omega_y \subseteq y + \big(B(S,\rho + \frac{1}{K}) \backslash B(S, \rho - \frac{1}{K})\big).\]
since $B(S,\rho)$ is regular, we conclude that
\[ \mathbb{P}_{y+B}(\Omega_y) \ll \frac{d}{\rho K}.\]
Inserting this into \eqref{previous-bound}, taking expectations, and using \eqref{efphi-general}, we conclude that
\[ \| F \|_{L^1(\mathbb{Z}/p\mathbb{Z})} + O( \frac{d}{\rho K} ) \gg \eta^C.\]
Now $d \ll \eta^{-C}$ and $\rho \gg \eta^C$. Thus by taking $K \geq C\eta^{C'}$ for large enough $C'$ we obtain the claim that $\Vert F \Vert_{L^1(\mathbb{Z}/p\mathbb{Z})} \gg \eta^C$.\end{proof}

Let $\mathcal{B}_{{\operatorname{triv}}}$ denote the rather trivial factor generated by the two atoms $[N]$ and $(\mathbb{Z}/p\mathbb{Z}) \setminus [N]$. With $f$ as in Theorem \ref{dens-inc} and this new notation we have $\delta 1_{[N]} = \mathbb{E}(f|\mathcal{B}_{\operatorname{triv}})$. Our next task is to iterate Theorem \ref{invert-u3-sigma}
via an energy increment argument to obtain the following structural result of ``Koopman-von Neumann'' type. The blueprint for arguments of this type is Szemer\'edi's proof of his \emph{regularity lemma} in graph theory \cite{szem:reg}. For other examples in additive combinatorics the reader might consult any of \cite{green-fin-field,green-montreal,green-tao-primes,green-tao-szem,tao-survey,taovu-book}.

\begin{theorem}[Quadratic Koopman-von Neumann theorem]\label{kvn}
Let $f: \mathbb{Z}/p\mathbb{Z} \to [-1,1]$ be a $1$-bounded function, and let $\eta > 0$.
Suppose also that $K$ is an integer such that $K \geq C \eta^{-C}$ for some sufficiently large constant $C > 0$.  Then there exists a
quadratic factor $(\mathcal{B}_1,\mathcal{B}_2)$ in $\mathbb{Z}/p\mathbb{Z}$ of complexity at most $(O( \eta^{-C} ), O( \eta^{-C} ))$
and resolution $K$ such that
\begin{equation}\label{efb-b1}
\|f-\mathbb{E}(f|\mathcal{B}_2 \vee \mathcal{B}_{\operatorname{triv}})\|_{U^3(\mathbb{Z}/p\mathbb{Z})} \leq \eta.
\end{equation}
\end{theorem}

\begin{proof} We run the following algorithm:
\begin{itemize}
\item Step 0: Initialize $\mathcal{B}_1 = \mathcal{B}_2 = \{ \emptyset, \mathbb{Z}/p\mathbb{Z}\}$. Thus $(\mathcal{B}_1,\mathcal{B}_2)$ is a quadratic factor with complexity $(0,0)$ and resolution $K$.
\item Step 1: If \eqref{efb-b1} holds, then \textsc{stop}.  Otherwise, apply Theorem \ref{invert-u3-sigma} with $f$ replaced\footnote{This function is bounded pointwise by 2. It is clear that a trivial rescaling of Theorem \ref{invert-u3-sigma} applies to such functions.} by $f - \mathbb{E}(f|\mathcal{B}_2 \vee \mathcal{B}_{\operatorname{triv}})$ to obtain a quadratic factor $(\mathcal{B}'_1, \mathcal{B}'_2)$ of complexity at most $(O( \eta^{-C} ), 1)$ and resolution $K$ such that
\begin{equation}\label{l1proj}
\| \mathbb{E}( f - \mathbb{E}(f|\mathcal{B}_2 \vee \mathcal{B}_{\operatorname{triv}}) | \mathcal{B}'_2 ) \|_{L^1(\mathbb{Z}/p\mathbb{Z})} \gg \eta^C.
\end{equation}
\item Step 2: Replace $(\mathcal{B}_1,\mathcal{B}_2)$ with $(\mathcal{B}'_1, \mathcal{B}_1 \vee \mathcal{B}'_2)$ (thus increasing the complexity of $(\mathcal{B}_1,\mathcal{B}_2)$ by at most $(O( \eta^{-C} ), 1)$),
and return to Step 1.
\end{itemize}

Observe from \eqref{l1proj} and Cauchy-Schwarz that
\[ \| \mathbb{E}( f - \mathbb{E}(f|\mathcal{B}_2 \vee \mathcal{B}_{\operatorname{triv}}) | \mathcal{B}'_2 ) \|_{L^2(\mathbb{Z}/p\mathbb{Z})} \gg \eta^C,\]
and hence
$$ \| \mathbb{E}( f - \mathbb{E}(f|\mathcal{B}_2 \vee \mathcal{B}_{\operatorname{triv}}) | \mathcal{B}_2 \vee \mathcal{B}'_2 \vee \mathcal{B}_{{\operatorname{triv}}} ) \|_{L^2(\mathbb{Z}/p\mathbb{Z})} \gg \eta^C.$$
By Pythagoras' theorem we conclude that
\[ \| \mathbb{E}(f | \mathcal{B}_2 \vee \mathcal{B}'_2 \vee \mathcal{B}_{\operatorname{triv}} ) \|_{L^2(\mathbb{Z}/p\mathbb{Z})}^2 - \| \mathbb{E}(f | \mathcal{B}_2 \vee \mathcal{B}_{\operatorname{triv}} ) \|_{L^2(\mathbb{Z}/p\mathbb{Z})}^2
\gg \eta^C.\]
It follows that every time we perform Step 2, the energy $\| \mathbb{E}(f | \mathcal{B}_2 \vee \mathcal{B}_{\operatorname{triv}} ) \|_{L^2(\mathbb{Z}/p\mathbb{Z})}^2$ increments by at least $\gg \eta^C$.
Since the energy is clearly bounded between $0$ and $1$, the algorithm can only run for at most $O( \eta^{-C})$ iterations, and the claim easily follows.
\end{proof}

If we apply this theorem (with $\eta := c \delta^4$ for some small $c$) and Lemma \ref{u3-lambda4} to the situation in Theorem \ref{dens-inc}
we obtain the following corollary.

\begin{corollary}[Anomalous AP4 count on a quadratic factor]\label{dens-cor}
Let the assumptions be as in Theorem \ref{dens-inc}.
Then there exists a quadratic factor $(\mathcal{B}_1,\mathcal{B}_2)$ in $\mathbb{Z}/p\mathbb{Z}$ of complexity at most $(O(\delta^{-C}), O(\delta^{-C}))$
and resolution $O( \delta^{-C})$
such that the function $g := \mathbb{E}(f|\mathcal{B}_2 \vee \mathcal{B}_{\operatorname{triv}})$ obeys
\begin{equation}\label{gap}
 |\Lambda(g,g,g,g) - \Lambda(\delta 1_{[N]}, \delta 1_{[N]}, \delta 1_{[N]}, \delta 1_{[N]})| \gg \delta^4.
 \end{equation}
\end{corollary}

Thus we have replaced the original function $f$ by the more structured function $g$.  Note that $\delta 1_{[N]} = \mathbb{E}(g|\mathcal{B}_{{\operatorname{triv}}})$. From this it is not hard to obtain, under the assumption that $f$ has an anomalous count of 4-term progressions, a substantial density increment for $f$
on a quadratic Bohr set.

\begin{corollary}[Density increment on quadratic Bohr set]\label{dens-cor2}
Let the assumptions be as in Theorem \ref{dens-inc}.
Then there exists a quadratic factor $(\mathcal{B}_1,\mathcal{B}_2)$ in $\mathbb{Z}/p\mathbb{Z}$ of complexity at most $(O(\delta^{-C}), O(\delta^{-C}))$
and resolution $O( \delta^{-C}) $, and an atom $B_2$ of $\mathcal{B}_2 \vee \mathcal{B}_{\operatorname{triv}}$ of density $\mathbb{P}_{\mathbb{Z}/p\mathbb{Z}}(B_2) \gg \exp( - O( \delta^{-C} ) )$
and contained in $[N]$ such that
\[ \mathbb{E}_{B_2}(f) \geq (1 + c) \delta\]
for some absolute contant $c > 0$.
\end{corollary}

\begin{proof}  Let $(\mathcal{B}_1,\mathcal{B}_2)$ and $g$ be as in Corollary \ref{dens-cor}. The facts that $[N]$ is measurable in $\mathcal{B}_2 \vee \mathcal{B}_{\operatorname{triv}}$ and that $f$ is supported on $[N]$ guarantee that $g$ is also supported on $[N]$.  Let $\Omega$ denote the set where $g \geq (1+c)\delta$, where $c > 0$ is a small constant to be chosen later, and
let $g' := (1-1_\Omega) g$. From Lemma \ref{l1-lambda4} we have
\[ |\Lambda(g,g,g,g) - \Lambda(g',g',g',g')| \leq 4 \mathbb{P}_{\mathbb{Z}/p\mathbb{Z}}(\Omega)\]
and
\[ |\Lambda(g',g',g',g') - \Lambda(\delta 1_{[N]}, \delta 1_{[N]}, \delta 1_{[N]}, \delta 1_{[N]})| \leq 8 \delta^3
\| g' - \delta 1_{[N]} \|_{L^1(\mathbb{Z}/p\mathbb{Z})}.\]
Furthermore we evidently have \[ \| g' - \delta 1_{[N]} \|_{L^1(\mathbb{Z}/p\mathbb{Z})} \leq \| g - \delta 1_{[N]} \|_{L^1(\mathbb{Z}/p\mathbb{Z})} + \mathbb{P}_{\mathbb{Z}/p\mathbb{Z}}(\Omega).\]  Combining
these three estimates together with \eqref{gap} we obtain
\[ \delta^3 \| g - \delta 1_{[N]} \|_{L^1(\mathbb{Z}/p\mathbb{Z})} + \mathbb{P}_{\mathbb{Z}/p\mathbb{Z}}(\Omega) \gg \delta^4.\]
Now observe that the positive part $(g - \delta 1_{[N]})_+$ of $g - \delta 1_{[N]}$ can only exceed $c\delta$ on $\Omega$, and hence has a total $L^1$ norm of
at most $c\delta + \mathbb{P}_{\mathbb{Z}/p\mathbb{Z}}(\Omega)$.  Since $g - \delta 1_{[N]}$ also has mean zero, we conclude that
\[\| g - \delta 1_{[N]} \|_{L^1(\mathbb{Z}/p\mathbb{Z})} = 2\| (g - \delta 1_{[N]})_+ \|_{L^1(\mathbb{Z}/p\mathbb{Z})} \ll c\delta + \mathbb{P}_{\mathbb{Z}/p\mathbb{Z}}(\Omega).\]
If $c$ is chosen small enough, we deduce that
\[ \mathbb{P}_{\mathbb{Z}/p\mathbb{Z}}(\Omega) \gg \delta^4.\]
Now $\mathcal{B}_2 \vee \mathcal{B}_{\operatorname{triv}}$ has complexity and resolution $O(\delta^{-C})$, and hence contains at most $\exp( O( \delta^{-C} ) )$ atoms. By the pigeonhole principle we can therefore find an atom $B_2$ of $\mathcal{B}_2 \vee \mathcal{B}_{\operatorname{triv}}$ which is contained in $\Omega$ and which has
$\mathbb{P}_{\mathbb{Z}/p\mathbb{Z}}(B_2) \gg \exp( - O( \delta^{-C} ) )$.  By construction we have $\mathbb{E}_{B_2}(f) \geq (1+c)\delta$, and the claim follows.\end{proof}

Our sole remaining task is to take this density increment for $f$ on a ``quadratic Bohr set'' and use it to obtain a similar density increment for $f$ on an arithmetic progression (Theorem \ref{dens-inc}). This we do by splitting the quadratic Bohr set into a union of progressions, a process we call \emph{linearisation}.

\section{Linearisation of quadratic Bohr sets}

We will decompose a quadratic Bohr set into a union of progressions. Our method for doing this does not naturally output progressions of equal sizes, and the following simple variant of the pigeonhole principle is designed to ensure that there is at least one progression which is quite long and on which $f$ has a substantial density increment.

\begin{lemma}[Pigeonhole principle]\label{dens-split}
Let $B$ be a non-empty set, and let $B = A_1 \cup \ldots \cup A_m$ be a partition of $B$ into $m$ disjoint sets.  Let $f: B \to \mathbb{R}^+$ be a $1$-bounded
nonnegative function.  Then for any $\varepsilon > 0$, there exists $i \in \{1,\dots,m\}$ such that $\mathbb{P}_B(A_i) > \varepsilon/m$ and
$$ \mathbb{E}_{A_i}(f) \geq \mathbb{E}_B(f) - \varepsilon.$$
\end{lemma}

\begin{proof}  Let $\Omega$ be the union of all the $A_i$ for which $\mathbb{P}_B(A_i) \leq \varepsilon/m$. We obviously have $\mathbb{P}_B(\Omega) \leq \varepsilon$.  From Bayes' identity and the fact that $0 \leq f \leq 1$ we have
\[ \mathbb{E}_B(f) = \mathbb{P}_B(\Omega) \mathbb{E}_\Omega(f) + (1-\mathbb{P}_B(\Omega)) \mathbb{E}_{B \backslash \Omega}(f) \leq \mathbb{P}_B(\Omega) + \mathbb{E}_{B \backslash \Omega}(f), \] and it follows that
\[ \mathbb{E}_{B \backslash \Omega}(f) \geq \mathbb{E}_B(f) - \varepsilon.\]
Partitioning $B \backslash \Omega$ into its constituent sets $A_i$, the claim follows from the usual pigeonhole principle.
\end{proof}

The next result provides the splitting of a quadratic Bohr sets into progressions, and is the main result of the section.

\begin{proposition}[Linearisation of quadratic Bohr sets]\label{linquad} Let $(\mathcal{B}_1,\mathcal{B}_2)$ in $\mathbb{Z}/p\mathbb{Z}$ of complexity at most $(d_1,d_2)$
and some resolution $K$, and let $B_2$ be an atom of $\mathcal{B}_2$.  Then one can partition $B_2 \cap [N]$ as the union of
$\ll d_2^{O(d_2)} N^{1 - c/(d_1+1)(d_2+1)^3}$ disjoint arithmetic progressions in $\mathbb{Z}/p\mathbb{Z}$.
\end{proposition}

\emph{Proof of Theorem \ref{dens-inc} assuming Proposition \ref{linquad}.} Suppose that $f$ satisfies the conditions of Theorem \ref{dens-inc}. By Corollary \ref{dens-cor2} we know that there is a quadratic factor $(\mathcal{B}_1,\mathcal{B}_2)$ in $\mathbb{Z}/p\mathbb{Z}$ of complexity at most $(O(\delta^{-C}), O(\delta^{-C}))$ and resolution $O(\delta^{-C})$, and an atom $B_2 \subseteq [N]$ of $\mathcal{B}_2 \vee \mathcal{B}_{\operatorname{triv}}$, having density at least $\exp(-O(\delta^{-C}))$ in $\mathbb{Z}/p\mathbb{Z}$, on which the average of $f$ is at least $(1 + c_0)\delta$. Using Proposition \ref{linquad} we may write $B_2$ as the union of $\exp(O(\delta^{-C})) N^{1 - c\delta^C}$ progressions. Taking $\varepsilon := c_0\delta/2$ in Lemma \ref{dens-split}, we obtain a progression of length at least $\exp(-O(\delta^{-C})) N^{c\delta^C}$ on which $f$ has average at least $(1 + \frac{1}{2}c_0)\delta$. To complete the proof of the theorem, we need to make sure that this length is in fact $\gg N^{c'\delta^{C'}}$ for absolute constants $c',C' > 0$. This may be ensured by taking the absolute constant in the condition \eqref{plarge} to be sufficiently large.\hfill{\usebox{\proofbox}}

It remains to prove Proposition \ref{linquad}.  We first deal with the linear component of the factor $(\mathcal{B}_1,\mathcal{B}_2)$.  Since $\mathcal{B}_2$ extends $\mathcal{B}_1$, there is a unique atom $B_1$ in $\mathcal{B}_1$ which contains $B_2$.

\begin{proposition}[Linearisation of linear Bohr sets]\label{linlin} Let $\mathcal{B}_1$ be a linear factor of complexity $d_1$ and resolution $K$. Let $B_1$ be an atom in $\mathcal{B}_1$. Then one can partition $B_1 \cap [N]$ as the union of $\ll 2^{d_1}N^{1 - 1/(d_1+1)}$ arithmetic progressions.
\end{proposition}

\begin{proof}  We can write $B_1$ as an uncentred Bohr set $B_\alpha(S, 1/2K)$, where $|S| \leq d_1$ and $\alpha \in (\mathbb{R}/\mathbb{Z})^S$.  Using the Kronecker approximation theorem (Proposition \ref{kat}) we can find a non-zero $r \in \mathbb{Z}/p\mathbb{Z}$ such that
\[ \| \xi r / p \|_{\mathbb{R}/\mathbb{Z}} \ll N^{-1/(d_1+1)} \]
for all $\xi \in S \cup \{1\}$.  If we then partition $\mathbb{Z}/p\mathbb{Z}$ into $O( N^{1 - 1/(d_1+1)} )$ arithmetic progressions of common difference $r$ and length
$O( N^{1/(d_1+1)} )$, we see that the intersection of each of these progressions with $B_1 \cap [N]$ will be the union of no more than $2^{d_1}$ smaller arithmetic
progressions, also of step $r$, and the claim follows.
\end{proof}

This last proposition improves our situation considerably, since it is much easier to understand quadratic phases on a progression than it is quadratic phases on a Bohr set.

\begin{proposition}[Linearisation of pure quadratic Bohr sets]\label{linpure}  Let $P$ be an arithmetic progression in $\mathbb{Z}/p\mathbb{Z}$, and let $\phi_1, \ldots, \phi_d: P \to \mathbb{R}/\mathbb{Z}$ be locally quadratic irrational phase functions on $P$. Consider the factor $\mathcal{B}_{\phi_1,K} \vee \ldots \vee \mathcal{B}_{\phi_d,K}$ of resolution $K$ defined by these phase functions \textup{(}cf. Definition \ref{def-52}\textup{)}. Then for any resolution $K$, every atom $B_2 \subseteq P$
of  can be partitioned as the union of $\ll d^{O(d)} |P|^{1 - c/(1+d)^3}$ disjoint arithmetic progressions.
\end{proposition}

Let us now see why Proposition \ref{linlin} and Proposition \ref{linpure} together imply Proposition \ref{linquad}.  First let us take all the progressions arising from Proposition \ref{linlin} which are rather small, say having length $O( N^{1/2(d_1+1)})$.  We can partition these progressions in the most trivial way into singletons, ending up with at most $O( N^{1 - 1/2(d_1+1)} )$ single-element
progressions in this way.  As for each longer progression $P$ in $B_1 \cap [N]$, of length $\gg N^{1/2(d_1+1)}$, we apply Proposition \ref{linpure} to see that $P \cap B_2$ is the union of $\ll d_2^{O(d_2)} |P|^{1 - c/(1+d_2)^3} \ll d_2^{O(d_2)} N^{-c/(1+d_1)(1+d_2)^3} |P|$ disjoint arithmetic progressions.  Assembling all of these progressions together as $P$ varies, we obtain $O( d_2^{O(d_2)} N^{1-c/(1+d_1)(1+d_2)^3})$ disjoint progressions in total, and Proposition \ref{linquad} follows.

It remains to prove Proposition \ref{linpure}. A result of this type, in which there is just a single quadratic phase, may be found in \cite{gowers-4-aps}. Here, however, we are dealing with $d$ quadratics rather than just one, and will have to take a little care to make sure that our exponents depend only polynomially on $d$ rather than exponentially. Because of this, we cannot, for example, simply iterate the analogous single-quadratic results from
\cite{gowers-4-aps}.  As a first step we may apply an affine linear transformation to $P$ and assume that $P = [1,M]$ for
some $M$, $1 \leq M \leq p$. We can also take $d \geq 1$ since the $d = 0$ case is trivial. It is easy to see, straight from the definition of a quadratic phase, that each $\phi_j: [1,M] \to \mathbb{R}/\mathbb{Z}$ takes the form
\[
\phi_j(n) = \alpha_j n^2 + \beta_j n + \gamma_j\]
for some $\alpha_j, \beta_j, \gamma_j \in \mathbb{R}/\mathbb{Z}$.  The set $B_2$ thus takes the form
$$ \{ n \in [1,M]: \| \alpha_j n^2 + \beta_j n + \gamma'_j \|_{\mathbb{R}/\mathbb{Z}} < 1/2K \hbox{ for } j = 1,\dots,d \}$$
where $\gamma'_j$ is some other element of $\mathbb{R}/\mathbb{Z}$.  Our objective is to partition this set into $\ll d^{O(d)} M^{1-c/d^3}$ disjoint arithmetic
progressions.

The first step, as in \cite{gowers-4-aps}, is to find a scale $r$ for which the effects of the quadratic components $\alpha_j n^2$ of each
phase are locally negligible.  Applying Proposition \ref{sqj} we can locate an integer $r$, $1 \leq r \leq \sqrt{M}$, such that
\begin{equation}\label{alphaj-quadratic}
\| \alpha_j r^2 \|_{\mathbb{R}/\mathbb{Z}} \ll d M^{-c_0/d^2}
\end{equation}
whenever $1 \leq j \leq d$, where $c_0 > 0$ is an absolute constant.
Now we can partition $[1,M]$ into at most $M^{1-c_0/4d^2}$ arithmetic progressions of step $r$ and lengths $\sim M^{c_0/4d^2}$ (that is, bounded above and below by absolute constants times this). It will suffice
to show that, for each such arithmetic progression $P$, the set $P \cap B_2$ can be partitioned into $\ll d^{O(d)} |P|^{1-1/2d}$ arithmetic progressions.

Let us fix one of these progressions $P = a, a+r, \ldots, a+(k-1)r$, where $k \sim M^{c_0/4d^2}$. From \eqref{alphaj-quadratic} we have
\begin{equation}\label{alphaj-quadratic2}
\| \alpha_j r^2 \|_{\mathbb{R}/\mathbb{Z}} \ll d k^{-4} \hbox{ for } j = 1,\dots,d.
\end{equation}
The set $P \cap B_2$ can be written as
$$ \{ a+ir: \| i^2 \alpha_j r^2 + \beta_{j,P} i + \gamma_{j,P} \|_{\mathbb{R}/\mathbb{Z}} < 1/2K \hbox{ for } j = 1,\dots,d \}$$
where $\beta_{j,P}, \gamma_{j,P}$ are some real numbers depending on $j$ and $P$.  Now we use Kronecker's theorem (Proposition \ref{kat}) to find a positive integer $s \leq \sqrt{k}$ such that
\begin{equation}\label{betajp-linear}
 \| \beta_{j,P} s \|_{\mathbb{R}/\mathbb{Z}} \ll k^{-1/2d}
 \end{equation}
for all $j \in \{1,\dots,d\}$.  We now partition $P$ into $\ll k^{1-1/2d}$ arithmetic progressions of step $rs$ and length $\ll k^{1/2d}$.  Consider a single such progression $Q$.  It can be written as
\[ Q = \{ a + (b + ts)r: 1 \leq t \leq T \}\]
for some $b \leq k$ and some $T \ll k^{1/2d}$, and its intersection with $B_2$ can be written as
\[ \{ a + (b + ts)r: 1 \leq t \leq T;
\| (b+ts)^2 \alpha_j r^2 + \beta_{j,P} (b+ts) + \gamma_{j,P} \|_{\mathbb{R}/\mathbb{Z}} < 1/2K \hbox{ for } j = 1,\dots,d\}.\]
The expression $(b+ts)^2 \alpha_j r^2 + \beta_{j,P} (b+ts) + \gamma_{j,P}$ can be rewritten (modulo $1$) as
$$ t^2 s^2 \{ \alpha_j r^2 \} + t (2bs \{ \alpha_j r^2 \} + \{ \beta_{j,P} s \}) + c_{j,P,Q}$$
where $\{x\} \in (-1/2,1/2]$ is the difference between $x$ and the nearest integer to $x$, and $c_{j,P,Q}$ is some real number.  Observe from \eqref{alphaj-quadratic2}, \eqref{betajp-linear} and the bounds $s \leq \sqrt{k}$, $b \leq k$, $T \ll k^{1/2d}$ that
the coefficients of $t^2$ and $t$ in this quadratic polynomial are $O(d/T^2)$ and $O(d/T)$ respectively.  Thus, for each fixed $j$, the set of values $t$ for which this expression has an $\mathbb{R}/\mathbb{Z}$ norm less than $1/2K$ is the union of $O(d)$ intervals (arithmetic progressions of step $1$).
This means that $Q \cap B_2$ is the union of at most $O(d)^d \ll d^{O(d)}$ intervals, and thus $P \cap B_2$ can be partitioned into
$\ll d^{O(d)} k^{1-1/2d}$ progressions as desired.  This concludes the proof of Proposition \ref{linpure} and hence, by earlier reductions, that of our main theorem.\hfill{\usebox{\proofbox}}

Number Theory (math.NT); Combinatorics (math.CO)


On the Hamming Distance between base-n representations of whole numbers

Abstract: We first introduce the Hamming distance between two strings. Then, we apply this concept to the representations of whole numbers in base n for all positive integers n > 2. We claim that a simple formula exists for the sum of all Hamming distances between pairs of consec- utive integers from 1 to m, which we will derive. We also state and prove other interesting results concerning the aforementioned topic.

\section{Introduction}
\paragraph{}
The remainder of this paper is organized as follows :
Section $2$ provides an introduction to the Hamming distance and its applications.
Some new results are stated with proof in Section $3$.
Section $4$ gives a Python implementation of the algorithm developed.. It also carries programs in Python for the same.
Finally, Section $5$ concludes the paper.

\section{Hamming Distance}
The Hamming distance is named after Richard Hamming. Defined between two strings of equal length, the Hamming distance gives the number of positions at which corresponding symbols differ. Alternatively, it measures the minimum number of substitutions required to change one string into the other, or the number of errors in transmission, not including insertions and deletions, required to transform one string into the other. In this paper, we will use the notation $H(S_1, S_2)$ to denote the Hamming distance between two strings of equal length - $S_1$ and $S_2$.
\newline
\newline
For example, the \verb|H(math, mats) = 1| while \verb|H(math, math) = 0|. Hamming distance is not defined between two strings of unequal length.
\newline
\newline
The following Python function calculates the Hamming distance, taking two strings of equal length as arguments :

\begin{verbatim}
def hammingdistance(s1, s2):
    assert len(s1) == len(s2)
    return sum(ch1 != ch2 for ch1, ch2 in zip(s1, s2))
\end{verbatim}

\section{Results}\label{results}
Let $n \in \mathbb{N}$  and $n \ge 2$. We consider the representations of whole numbers in base $n$. We append zeros to the left of the numerals in case the strings are of unequal length. This happens between $m-1$ and $m$ if and only if $m$ is a power of $n$.
\newline
\newline
{\underline {Lemma}} : $H(m, m-1) = l+1$ where $l$ is the exponent of the highest power of $n$ that divides $m$.
\newline
\newline
{\underline {Proof}} : We prove this by considering $3$ cases.
\newline
\newline
Case $1$ : $m = n^l$ for some $l \in \mathbb{N}$
\newline
\newline
Here, $m = {(100...000)}_n$ ($1$ followed by $l$ zeroes).
Let A be the digit in base $n$ that represents $n - 1$. So, $m - 1 =$ AAA...AAA ($l$ A digits).

Clearly, $H(m, m-1) = l + 1$ and we are done.
\newline
\newline
Case $2$ : $m = pn^q$ for some $q \in \mathbb{N}$ where $q \ge 1$ and $(p, n) = 1$. Let,

\begin{equation}
m = {n^k}{a_k} + {n^{k-1}}{a_{k-1}} + ... + {n^2}{a_2} + {n^1}{a_1} + {n^0}{a_0}
\end{equation}

Trivially, all $a_j$ for $j < q$ will be zeros.

And,

\begin{equation}
m - 1 = {n^k}{b_k} + {n^{k-1}}{b_{k-1}} + ... + {n^2}{b_2} + {n^1}{b_1} + {n^0}{b_0}
\end{equation}

Here, $a_0 = 0$ as $n$ divides $m$.

Now, $m - 1$ will have all $b_j = (n-1)$ for $j < q$ and $b_q = a_q - 1$. For $i > q$, on the other hand, $a_i = b_i$.
\newline
\newline
Thus, $H(m, m-1) = q + 1$ and we are done.
\newline
\newline
Case $3$ : $n$ does not divide $m$. Let
\begin{equation}
m = {n^k}{a_k} + {n^{k-1}}{a_{k-1}} + ... + {n^2}{a_2} + {n^1}{a_1} + {n^0}{a_0}
\end{equation}
\begin{equation}
m = \sum\limits_{i=0}^k {n^i}{a_i}
\end{equation}

Similarly,

\begin{equation}
m - 1 = {n^k}{b_k} + {n^{k-1}}{b_{k-1}} + ... + {n^2}{b_2} + {n^1}{b_1} + {n^0}{b_0}
\end{equation}
\begin{equation}
m - 1 = \sum\limits_{i=0}^k {n^i}{b_i}
\end{equation}
Since $n$ does not divide $m$, $a_0 \geq 1$. Therefore $b_0$, the remainder of $m - 1$ upon division of $n$, is simply $a_0 - 1$, and $b_i = a_i$ for all $i \geq 1$. Therefore $H(m, m-1) = 1$ and we are done. \hfill${\blacksquare}$
\newline
\newline

Now that we have found a way to calculate the Hamming distance between two consecutive numbers in base $n$, we will derive a formula for the sum of all such Hamming Distances up to a given $m$. Let
\begin{equation}
S(m) = H(0, 1) + H(1, 2) + H(2, 3) ... + H(k-2, k - 1) + H(k - 1, k)
\end{equation}
\begin{equation}
S(m) = \sum\limits_{i=0}^{k-1} H(i, i+1)
\end{equation}

From the lemma, we know that $H(m, m-1) = l+1$ where $l$ is the exponent of the highest power of $n$ that divides $m$.

Thus,
\begin{equation}
S = \sum\limits_{i=1}^{m} (P_i + 1)
\end{equation}

Where $P_i$ represents the highest power of $n$ contained in $i$. Trivially, the least value that $P_i + 1$ will ever attain for any $i$ is $1$. Using Iverson Brackets,
\newline
\begin{equation}
S(m) = \sum_{i=1}^m \sum_{j=0}^\infty [n^j | i]
\end{equation}

which we can then rewrite as, by exchanging the order of summation,

\begin{equation}
S(m) = \sum_{j=0}^\infty \sum_{i=1}^m [n^j | i]
\end{equation}

But there are $\lceil m/n^j \rceil$ integers from 1 to $n$ that divide $n^j$, so

\begin{equation}
S(m) = \sum_{j=0}^\infty \lceil m/n^j \rceil
\end{equation}

Finally, we only need to take the sum up to $\lceil \log_n m \rceil$, since $m < n^j$ for $j > \log_n m$.
\newline

And we are done. \hfill $\blacksquare$

\section{Implementation}
The following Python code finds the sum of Hamming distances in base $n$ upto a given $m$ :

\begin{verbatim}
i = int(raw_input('Enter a number : '))
n = int(raw_input('Enter a base : '))

i = int(i, n)

s = 0
x = 0

def hamming_distance(s1, s2):
    assert len(s1) == len(s2)
    return sum(ch1 != ch2 for ch1, ch2 in zip(s1, s2))

while (x < i):
    s1 = base10toN(x, n)
    s2 = base10toN(x+1, n)
    if (len(s1) != len(s2)):
         s1 = '0' + s1
    s = s + hamming_distance(s1, s2)
    x = x + 1

print s
\end{verbatim}

The following Python code prints the sum of all Hamming distances from 0 to m in base n using the results proved in this paper :

\begin{verbatim}
m = int(raw_input('Enter a number : '))
n = int(raw_input('Enter a base : '))

j = i
h = 0
while (j > 1):
    j = j / n
    h = h + 1

s = 0
while (h >= 0):
    s = s + ( i / (n ** h) )
    h = h - 1

print s
\end{verbatim}

\section{Conclusions}
We conclude that the sum of the Hamming distances of consecutive integers is given by a simple formula and can be computed easily and efficiently.

\section{References}
\begin{enumerate}
\item Hamming, R. W. Error detecting and error correcting codes. Bell System Tech. J. 29, (1950). 147 - 160.
\item Public Domain Material, Federal Standard 1037C
\end{enumerate}

\bibliographystyle{abbrv}