On the connection between the conjugate gradient method and quasi-Newton methods on quadratic problems

Abstract: It is well known that the conjugate gradient method and a quasi-Newton method, using any well-defined update matrix from the one-parameter Broyden family of updates, produce identical iterates on a quadratic problem with positive-definite Hessian. This equivalence does not hold for any quasi-Newton method. We define precisely the conditions on the update matrix in the quasi-Newton method that give rise to this behavior. We show that the crucial facts are, that the range of each update matrix lies in the last two dimensions of the Krylov subspaces defined by the conjugate gradient method and that the quasi-Newton condition is satisfied. In the framework based on a sufficient condition to obtain mutually conjugate search directions, we show that the one-parameter Broyden family is complete.A one-to-one correspondence between the Broyden parameter and the non-zero scaling of the search direction obtained from the corresponding quasi-Newton method compared to the one obtained in the conjugate gradient method is derived. In addition, we show that the update matrices from the one-parameter Broyden family are almost always well-defined on a quadratic problem with positive-definite Hessian. The only exception is when the symmetric rank-one update is used and the unit steplength is taken in the same iteration. In this case it is the Broyden parameter that becomes undefined.

\section{Introduction}
\label{sec:introduction}
In this paper we examine some well-known methods used for solving unconstrained optimization
problems and specifically their behavior on quadratic problems. A
motivation why these problems are of interest is that the task of solving a linear system of equations
$Ax=b$, with the assumption $A=A^T \succ 0$, may equivalently be considered as the
one of solving an unconstrained quadratic programming problem,
\begin{equation}\label{qp}
\min_{x \in \mathbb{R}^n}q(x)=\min_{x \in \mathbb{R}^n}\frac{1}{2}x^THx+c^Tx,\tag{QP}
\end{equation}
where one lets $H=A$ and $c=\nolinebreak-b$ to obtain the usual notation.

Given an initial guess $x_0$, the general
idea of most methods for solving \eqref{qp} is to, in each iteration $k$, generate a search direction $p_k$ and
then take a steplength $\alpha_k$ along that direction to approach the
optimal solution. For $k \geq 0$, the next iterate is hence obtained as
\begin{equation}\label{nextiterate}
x_{k+1}=x_k+\alpha_kp_k.
\end{equation}
The main difference between methods is the way the search
direction $p_k$ is generated. For high-dimensional problems it is
preferred that only function and gradient values are used in
calculations. The gradient of the objective function
$q(x)$ is given by $g(x)=Hx+c$ and its value at $x_k$ is denoted by
$g_k$.

The research presented in this paper stems from the desire to better
understand the well-known connection between the \emph{conjugate
gradient method}, henceforth CG, and \emph{quasi-Newton methods},
henceforth QN. We are interested in determining precise conditions in
association with the generation of $p_k$ in QN such that, using
exact linesearch, CG and QN will generate the same sequence
of iterates as they approach the optimal solution of \eqref{qp}.

CG and QN are introduced briefly in Section \ref{sec:background} where we also state some
background results on the connection between
these two methods. In Section \ref{sec:results} we present our results and some concluding remarks
are made in Section \ref{sec:conclusion}.

\section{Background}
\label{sec:background}
On \eqref{qp}, naturally, we consider the use of \emph{exact
  linesearch}. Then in iteration $k$, the optimal steplength is given by
\begin{equation}\label{steplength}
\alpha_k=-\frac{p_k^Tg_k}{p_k^TH p_k}.
\end{equation}
Since $H=H^T \succ 0$, it can be shown that the descent property,
$q(x_{k+1}) < q(x_{k})$, holds, as long as
\begin{equation}\label{nonorth}
p_k^Tg_k \neq 0.
\end{equation}
Note that the usual requirement $p_k^Tg_k<0$, i.e. that is $p_k$ is a \emph{descent
  direction} with respect to $q(x)$ at $x_k$ with steplength $\alpha_k >0$, is included in
the more general statement of \eqref{nonorth}.

Given two parallel vectors and an initial point $x_k$, performing exact
linesearch from the initial point with respect to a given objective function along these two
vectors will yield the same iterate $x_{k+1}$. Hence, two methods will find the
same sequence of iterates if and only if the search directions generated
by the two methods are parallel. Therefore, in the remainder of this
paper our focus will be on parallel search directions rather than
identical iterates. We will denote parallel vectors by $p  \ / \! / \  p'$.

On \eqref{qp}, one is generally interested in methods for which
the optimal solution is found in at most $n$ iterations. It can be
shown that a sufficient property for this behavior is
that the method generates search directions which are
mutually conjugate with respect to $H$, i.e. $p_i^THp_j=0,
\forall i \neq j$, see, e.g., \cite[Chapter 5]{nocedalwright}.

\subsection{Conjugate gradient method} \label{sec:cg}
A generic way to generate conjugate vectors is by means of the
conjugate Gram-Schmidt process. Given a set of linearly
independent vectors $\{a_0, \dots, a_{n-1}\}$,
a set of vectors $\{p_0, \dots,p_{n-1}\}$ mutually conjugate with
respect to $H$ can be constructed by letting $p_0=a_0$
and for $k>0$,
\begin{equation}\label{cdm}
p_k=a_k+\sum_{j=0}^{k-1}\beta_{kj}p_j.
\end{equation}
The values of $\{\beta_{kj}\}_{j=0}^{k-1}$ are uniquely determined in order to make $p_k$
conjugate to $\{p_0, \dots, p_{k-1}\}$ with respect to $H$. \emph{Conjugate
direction methods} is the common name for all methods which
are based on generating search directions in this manner.

With the choice $a_k=-g_k$ in \eqref{cdm} one obtains the conjugate gradient
method, CG, of Hestenes and Stiefel \cite{HestenesStiefel}. In effect, in CG let $p_0=-g_0$, and for $k>0$ the only $\beta$-value in \eqref {cdm} that will be non-zero is
\begin{equation}\label{beta}
\beta_{k,k-1}=\frac{p_{k-1}^THg_k}{p_{k-1}^TH p_{k-1}},
\end{equation}
where one may drop the first sub-index. It can be shown that
$\beta_{k-1}=g_k^Tg_k/g_{k-1}^Tg_{k-1}$, so that \eqref{cdm} may be
written as
\begin{equation}\label{cgdir2}
p_k=-g_k+\beta_{k-1}p_{k-1}=-\sum_{i=0}^k\frac{g_k^Tg_k}{g_i^Tg_i}g_i.
\end{equation}
From the use of exact linesearch it holds that $g_k^Tp_j=0$, for all
$j\leq k-1$, which implies that $g_k^Tg_j=0$, for all
$ j\leq k-1$, i.e. the produced gradients are orthogonal and therefore linearly
independent, as required. As for any conjugate
directions method using exact linesearch, the search
directions of CG are descent directions.   See,
e.g., \cite{cgwopain} for an intuitive introduction to the
conjugate gradient method with derivation of the above relations.

In CG, one only needs to
store the most recent previous search direction, $p_{k-1}$. This is a reduction in
the amount of storage required compared to a general conjugate direction
method where potentially all previous search directions are needed to
compute $p_k$.

Although equations \eqref{nextiterate}, \eqref{steplength},
\eqref{beta} and \eqref{cgdir2} give a complete description of an
iteration of CG, the
power and richness of the method is somewhat clouded in notation. An
intuitive way to picture what happens in an iteration of CG is to
describe it as a Krylov subspace method.

\begin{definition}\label{krylov}
Given a matrix $A$ and a vector $b$ the \emph{Krylov subspace}
generated by $A$ and $b$ is
given by $\mathcal{K}_{k}(b,A) =span\{ b, Ab, \ldots, A^{k-1}b \}$.
\end{definition}
Krylov subspaces are linear subspaces, which are expanding,
i.e. $\mathcal{K}_{1}(b, A) \subseteq \mathcal{K}_{2}(b, A) \subseteq
\mathcal{K}_{3}(b, A) \subseteq
\dots$, and $\dim(\mathcal{K}_{k}(b,A))=k$. Given $x \in
\mathcal{K}_{k}(b, A)$, then $Ax \in
\mathcal{K}_{k+1}(b, A)$, see, e.g., \cite{gutknecht} for an introduction to
Krylov space methods. CG is a Krylov subspace method and
iteration $k$ may be formulated as the following constrained
optimization problem
\begin{equation}\label{qpcg}
\min \ q(x), \ \text{s.t.} \ x \in x_0+\mathcal{K}_{k+1}(p_0,H),\tag{CG$_k$}
\end{equation}
see, e.g., \cite[Chapter 5]{nocedalwright}. The optimal solution of (CG$_k$) is $x_{k+1}$ and the corresponding multiplier is
given by \nolinebreak{$g_{k+1}=\nabla q(x_{k+1})=Hx_{k+1}+c$.} In each iteration, the
dimension of the affine subspace where the optimal solution is sought
increases by one. After at most $n$ iterations the optimal solution in
$\mathbb{R}^n$ is found, which will
then be the optimal solution of (QP).\footnote{It may happen, depending on the
number of distinct eigenvalues of $H$ and the orientation of $p_0$,
that the optimal solution is found after less than $n$ iterations, see, e.g.,
\cite[Chapter 6]{saad}.}

The search direction $p_k$ belongs to $\mathcal{K}_{k+1}(p_0,H)$, and as it is conjugate to all
previous search directions with respect to $H$ it holds that $span\{p_0, p_1, \dots, p_k\}=\mathcal{K}_{k+1}(p_0, H)$,
i.e., the search directions $p_0, \dots, p_k$ form an $H$-orthogonal
basis for $\mathcal{K}_{k+1}(p_0,H)$. We will henceforth refer to the search direction produced by CG, in
iteration $k$ on a given \eqref{qp}, as $p_k^{CG}$.

Since the gradients are mutually orthogonal, and because of the relationship
with the search directions in \eqref{cgdir2}, it can be shown that $span\{g_0,
\dots, g_k\}=\mathcal{K}_{k+1}(p_0,H)$, i.e. the gradients form an orthogonal
basis for $\mathcal{K}_{k+1}(p_0,H)$.

General conjugate direction methods can not be described as
Krylov subspace methods, since in general $span\{p_0, \dots,
p_{k}\} \neq \nolinebreak\mathcal{K}_{k+1}(p_0,H)$. We will use this
special characteristic of CG when
investigating the connection to QN.

Although our focus is quadratic programming it deserves mentioning that CG was extended to general unconstrained problems by
Fletcher and Reeves \cite{fletcherreeves}.

\subsection{Quasi-Newton methods}\label{sec:qn}

In QN methods the search directions are generated by solving
\begin{equation}\label{calp}
B_kp_k=-g_k,
\end{equation}
in each iteration $k$, where the matrix $B_k$ is chosen to be an approximation of $H$ in
some sense.\footnote{The
choice $B_k=H$ would give Newton's method, whereas the choice $B_k=I$
would give the steepest-descent method.} In this paper, we will consider \emph{symmetric} approximations of the Hessian,
i.e. $B_k=B_k^T$. It is also possible to consider unsymmetric
approximation matrices, see, e.g., \cite{huang}.

The first suggestion of a QN method was made by Davidon in 1959
\cite{davidon}, using the term \emph{variable metric method}. In
1963, in a famous paper by Fletcher and Powell \cite{fletcherpowell}, Davidon's
method was modified\footnote{"We have made both a simplification by
  which certain orthogonality conditions which are important to the
  rate of attaining the solution are preserved, and also an
  improvement in the criterion of convergence.'' \cite{fletcherpowell}} and this was the starting point for making
these QN methods widely known, used and studied.

We choose to work with an approximation of the Hessian rather than
an approximation of the inverse Hessian, $M_k$, as many of the earlier papers
did, e.g. \cite{fletcherpowell}. Our results can however straightforwardly be derived
for the inverse point of view where \eqref{calp} is replaced by the
equation $p_k=-M_kg_k$.

The approximation matrix $B_k$ used in iteration $k$ to solve for $p_k$ is obtained by adding an
\emph{update matrix}, $U_k$, to the previous approximation matrix,
\begin{equation}\label{Bupdate}
B_k=B_{k-1}+U_k.
\end{equation}

One often considers the Cholesky factorization of $B_k$, then
\eqref{calp} can be solved in order of $n^2$ operations. Also, if in
\eqref{Bupdate} the update matrix $U_k$ is of low-rank, one does not need to compute the Cholesky factorization
of $B_k$ from scratch in each iteration, see, e.g., \cite{practicalopt}.

One of the most well-known update schemes is the one using update
matrices from the \emph{one-parameter Broyden
family of updates} \cite{broyden1967} described by
\begin{equation}\label{broyden}
U_k= \frac{Hp_{k-1}p_{k-1}^TH}{p_{k-1}^T Hp_{k-1}}-\frac{B_{k-1}p_{k-1}p_{k-1}^TB_{k-1}}{p_{k-1}^T B_{k-1}p_{k-1}}+\phi_k p_{k-1}^TB_{k-1}p_{k-1} ww^T,
\end{equation}
with
$$
w=\frac{Hp_{k-1}}{p_{k-1}^T Hp_{k-1}}-\frac{B_{k-1}p_{k-1}}{p_{k-1}^T B_{k-1}p_{k-1}},
$$
and where $\phi_k$ is a free parameter, known as the
\emph{Broyden parameter}.
Equation \eqref{broyden} may be written more compactly, momentarily dropping all
subscripts, as
\begin{equation}\label{broyden2}
U=\left(
\begin{array}{ll}
\frac{1}{p^T Hp}Hp & \frac{1}{p^T Bp}Bp
\end{array} \right)
\left(
\begin{array}{cc}
p^THp+\varphi & -\varphi \\
-\varphi  & -p^TBp+\varphi
\end{array} \right)
\left(
\begin{array}{c}
\frac{1}{p^T Hp}p^TH\\
\frac{1}{p^T Bp}p^TB
\end{array} \right),
\end{equation}
where $\varphi=\phi_k p^TBp$. It is common to express the
one-parameter Broyden family in terms of $y_k=g_{k+1}-g_{k}$
and $s_k=x_{k+1}-x_{k}$, see, e.g. \cite{fletcherpractical}, but as our interest is in the search directions
we prefer the equivalent form of \eqref{broyden}.

For all updates in this family, \eqref{Bupdate} has the property of
\emph{hereditary symmetry}, i.e. if $B_{k-1}$ is symmetric then $B_k$
will be symmetric. The update given by the choice $\phi_k=0$ is known as
the \emph{Broyden-Fletcher-Goldfarb-Shanno}-update, or BFGS-update
for short. For this update, when exact linesearch is used, \eqref{Bupdate} has the property of \emph{hereditary
  positive definiteness}, i.e. if $B_{k-1} \succ 0$ then $B_{k} \succ
0$. An implication of this is that for all
updates given by $\phi_k \geq 0$, when exact linesearch is used, \eqref{Bupdate} has the property of hereditary
  positive definiteness, see,
e.g., \cite[Chapter 9]{luenberger}. Note that there are updates in the
one-parameter Broyden family for which \eqref{Bupdate} does not have
this property.

\subsection{Background results}\label{sec:background results}
In \cite{dixon}, Dixon has shown that on any smooth function,
using perfect linesearch\footnote{A generalization of exact
  linesearch for general smooth functions, see \cite{dixon}.}, the one-parameter Broyden family gives rise
to parallel search directions. On \eqref{qp}, these
search directions will in addition be mutually conjugate with respect to
$H$, see, e.g., \cite{huang}.

On \eqref{qp} it is well-known that these conjugate search directions,
generated by the one-parameter Broyden family, will be parallel to
those of CG, i.e. $p_k  \ / \! / \  p_k^{CG}$, for all $k$. See, e.g.,
\cite{fletcherpractical, huang, nazareth}. In particular Theorem 3.4.2
on page 65 of \cite{fletcherpractical} states that CG and QN using
well-defined update matrices from the one-parameter Broyden family
generate identical iterates. Note that this connection between CG and
QN does not hold for general convex functions.

In this paper we approach the well-known connection between QN
and CG from another perspective. The main question handled in this
paper is: when solving \eqref{qp}, what are the precise conditions on
$B_k$ and $U_k$ such that $p_k  \ / \! / \  p_k^{CG}$ is obtained? We
provide an answer by turning our attention to the search directions
defined by CG and the Krylov subspaces they span. In
Proposition~\ref{U_cg_2} we state explicit conditions on $U_k$ and in
Theorem~\ref{thmbroyden} we extend the well-known connection between
CG and QN as we show that, under a sufficient condition to generate
conjugate search directions, no other update matrices than those in
the one-parameter Broyden family will make $p_k  \ / \! / \  p_k^{CG}$.

\section{Results}\label{sec:results}
As a reminder for the reader we
will, in the following proposition, state the necessary and sufficient
conditions for a vector $p_k$ to be parallel to the vector
$p_k^{CG}$ given that $p_i  \ / \! / \
p_i^{CG}$, for all $i \leq k-1$. Given $x_0$, one may calculate $g_0=Hx_0+c$.
\begin{proposition}\label{p_cg}
Let $p_0=p_0^{CG}=-g_0$ and $p_i  \ / \! / \
p_i^{CG}$, for all $i \leq k-1$. Assume $p_k \neq 0$, then $p_k  \ / \! / \  p_k^{CG}$ if and only if
\begin{itemize}
\item[(i)] $p_k \in \mathcal{K}_{k+1}(p_0,H)$, and,
\item[(ii)] $p_k^THp_i=0, \quad  \forall i \leq k-1$.
\end{itemize}
\end{proposition}
\begin{proof}
For the sake of completeness we include the proof.

 \emph{Necessary:} Suppose $p_k$ is parallel to $p_k^{CG}$.
 Then $p_k=\delta_k p_k^{CG}$, for an arbitrary nonzero
  scalar $\delta_k$.  As $p_k^{CG}$
  satisfies (i), it holds that
$$
p_k=\delta_k p_k^{CG} \in  \mathcal{K}_{k+1}(p_0,H),
$$
since $ \mathcal{K}_{k+1}(p_0,H)$ is a
  linear subspace.
And as $p_k^{CG}$ satisfies (ii), it follows that,
$$
p_k^THp_i=\delta_k \delta_i (p_k^{CG})^TH p_i^{CG}=0, \quad \forall i
\leq k-1.
$$

\emph{Sufficient:} Suppose $p_k$ satisfies (i) and (ii). The set of
vectors $\{p_0^{CG}, \dots, p_{k}^{CG}\}$ form an $H$-orthogonal basis for the
space $\mathcal{K}_{k+1}(p_0,H)$ and the set of vectors $\{p_0, \dots, p_{k-1}\}$ form an $H$-orthogonal basis for the
space $\mathcal{K}_{k}(p_0,H)$. Since $p_k$ satisfies (i) and (ii) it
must hold that $p_k  \ / \! / \  p_k^{CG}$.
\end{proof}

These necessary and sufficient conditions will serve as a foundation for the rest of
our results. We will determine conditions, first on $B_k$, and
second on $U_k$ used in iteration $k$ of QN, in order for $p_k  \ / \! / \  p_k^{CG}$.

Given the current iterate $x_k$, one may calculate
$g_k=Hx_k+c$. Hence, in iteration $k$ of QN, $p_k$ is determined from
\eqref{calp} and depends entirely
on the choice of $B_k$. We make the assumption that \eqref{calp} is
compatible, i.e. that a solution exists. A well-known sufficient condition for $p_k$ to satisfy (ii) of
Proposition \ref{p_cg} is for $B_k$ to satisfy
\begin{equation}\label{conj}
B_kp_i=H p_i, \quad \forall i \leq k-1.
\end{equation}
This condition, known as the \emph{hereditary condition}, will be used in the following results, which implies that the stated conditions on $B_k$ and later
$U_k$ will be \emph{sufficient} conditions, and not necessary and sufficient
as those in Proposition \ref{p_cg}. All $B_k$ satisfying \eqref{conj} can be seen as defining
different conjugate direction methods, but only certain choices of
$B_k$ will generate conjugate directions parallel to those of CG.

Assume that $B_k$ is constructed as,
\begin{equation}\label{Bconst}
B_k=I +V_k,
\end{equation}
i.e. an identity matrix plus a matrix $V_k$.\footnote{Any symmetric
  matrix can be expressed like this.} We make no assumptions on $V_k$ except symmetry.

Let $B_0=I$, then $p_0=p_0^{CG}=-g_0$. Given $p_i  \ / \! / \
p_i^{CG}$, for all $i \leq k-1$, the
following lemma gives sufficient
conditions on $B_k$ in order for $p_k  \ / \! / \  p_k^{CG}$.

\begin{lemma}\label{B_cg}
Let $B_0=I$, so that $p_0=p_0^{CG}=-g_0$.  Assume that $p_i  \ / \! / \
p_i^{CG}$, for all $i \leq k-1$, and that a solution $p_k$ is obtained from
\eqref{calp}. Then $p_k  \ / \! / \  p_k^{CG}$ if $B_k$ satisfies
\begin{itemize}
\item[(i)] $\mathcal{R}(V_k) \subseteq  \mathcal{K}_{k+1}(p_0,H)$, and,
\item[(ii)] $B_kp_i=Hp_i, \quad \forall i \leq k-1$,
\end{itemize}
where $\mathcal{R}(V_k)=\{y\ : \ y=V_kx\}$, the \emph{range-space} of
the matrix $V_k$.

\end{lemma}
\begin{proof}
First we show that $p_k$ satisfies condition (i) of Proposition \ref{p_cg}. If
\eqref{Bconst} is inserted into \eqref{calp} one obtains,
$$
\big(I +V_k\big)p_k=-g_k,
$$
so that
$$
p_k=-g_k-V_kp_k.
$$
Since $g_k \in \mathcal{K}_{k+1}(p_0,H)$,
and $\mathcal{R}(V_k) \subseteq \mathcal{K}_{k+1}(p_0,H)$, it holds that $p_k \in \mathcal{K}_{k+1}(p_0,H)$. Hence, $p_k$
satisfies condition (i) of Proposition \ref{p_cg}. Secondly, $p_k$ satisfies condition (ii) of Proposition \ref{p_cg}
since condition (ii) of this lemma is identical to \eqref{conj}.

Hence, $p_k  \ / \! / \  p_k^{CG}$ by Proposition \ref{p_cg}.
\end{proof}

As $B_k$ is updated according to \eqref{Bupdate}, one would prefer to
have conditions, in iteration $k$, on the update matrix $U_k$ instead of on the
entire matrix $B_k$. Therefore, we now modify Lemma \ref{B_cg} by
noting that equation \eqref{Bconst} may be stated as
\begin{equation}\label{Uconst}
B_k=B_{k-1}+U_k=I+V_{k-1} +U_k.
\end{equation}
We make no
assumptions on $U_k$ except symmetry.
Note that one
may split \eqref{conj}  as
\begin{equation}\label{conjrest}
B_kp_{i}=Hp_{i},  \quad \forall i \leq k-2,
\end{equation}
\begin{equation}\label{conjqn}
B_kp_{k-1}=Hp_{k-1}.
\end{equation}
Equation \eqref{conjqn} is known as the \emph{quasi-Newton
  condition}.\footnote{In the literature, much emphasis is placed on that updates
should satisfy \eqref{conjqn}. This condition alone is
  not a sufficient condition on $B_k$ to give conjugate
  directions. } Using \eqref{Bupdate} one can reformulate \eqref{conjrest} and \eqref{conjqn} in terms of
$U_k$, see (ii) and (iii) of the following proposition.

Given $B_{k-1}$ that satisfies condition (i) of
Lemma \ref{B_cg}, and given $p_i  \ / \! / \
p_i^{CG}$, for all $i \leq k-1$, the
following proposition gives sufficient
conditions on $U_k$ in order for $p_k  \ / \! / \  p_k^{CG}$.

\begin{proposition}\label{U_cg_2}
Let $B_0=I$, so that $p_0=p_0^{CG}=-g_0$. Assume that $p_i  \ / \! / \
p_i^{CG}$, for all $i \leq k-1$, that $B_{k-1}$ satisfies condition
(i) of Lemma \ref{B_cg}, that a solution $p_k$ is obtained from
\eqref{calp} and that $B_k$ is obtained from \eqref{Uconst}. Then
$p_k  \ / \! / \  p_k^{CG}$  if $U_k$ satisfies
\begin{itemize}
\item[(i)] $\mathcal{R}(U_k) \subseteq  \mathcal{K}_{k+1}(p_0,H)$,
\item[(ii)] $U_k p_i=0, \quad  \forall i \leq k-2$, and,
\item[(iii)] $U_kp_{k-1}=(H-B_{k-1})p_{k-1}$.
\end{itemize}
\end{proposition}

\begin{proof}
The proof is identical to the one of Lemma \ref{B_cg}. If
\eqref{Uconst} is inserted into \eqref{calp} one obtains
$$
p_k=-g_k-V_{k-1}p_k-U_kp_k.
$$

By condition (i) of Lemma \ref{B_cg}, $\mathcal{R}(V_{k-1}) \subseteq \mathcal{K}_{k}(p_0,H)$, and since \linebreak$\mathcal{K}_{k}(p_0,H) \subseteq
\mathcal{K}_{k+1}(p_0,H)$, it follows that $V_{k-1}p_k \in \mathcal{K}_{k+1}(p_0,H)$.
Therefore, since $g_k \in \mathcal{K}_{k+1}(p_0,H)$ and $\mathcal{R}(U_k) \subseteq
\mathcal{K}_{k+1}(p_0,H)$, it holds that $p_k \in
\mathcal{K}_{k+1}(p_0,H)$. Hence, $p_k$ satisfies condition (i) of Proposition \ref{p_cg}.

Conditions (ii) and (iii) of this lemma are merely a reformulation of \eqref{conj}, hence $p_k$ satisfies condition (ii) of Proposition \ref{p_cg}.

Hence, $p_k  \ / \! / \  p_k^{CG}$ by Proposition \ref{p_cg}.
\end{proof}

The assumption of the previous proposition, that $B_{k-1}$ is chosen to satisfy condition (i) of Lemma \ref{B_cg},
will be satisfied if, in each iteration $k$, the update matrix $U_k$ is chosen
according to conditions (i)-(iii) of Proposition \ref{U_cg_2}. This is summarized in the
following corollary.

\begin{corollary}
If in each iteration $k$ the update matrix $U_k$ is chosen to satisfy
conditions (i)-(iii) of Proposition \ref{U_cg_2}, then $p_k  \ / \! / \
p_k^{CG}$ for all $k$.
\end{corollary}

Next we state a result, which will be needed in our further
investigation, and holds for any update scheme of $B_k$, which generates search
directions parallel to those of CG.

\begin{proposition}\label{Bdefined}
If $p_k  \ / \! / \
p_k^{CG}$ for all $k$ and a solution $p_k$ obtained from
\eqref{calp}, then $p_{k}^TB_{k}p_{k} \neq
0$, unless $B_{k}p_{k}=-g_{k}=\nolinebreak 0$.
\end{proposition}
\begin{proof}
Since $p_k=\delta_k p_k^{CG}$, for some non-zero scalar $\delta_k$,
and since $p_k^{CG}$ is a descent direction it holds that
$$
p_{k}^TB_{k}p_{k}=-p^T_{k}g_{k}=-\delta_k (p_k^{CG})^Tg_k \neq 0,
$$
for some non-zero scalar $\delta_k$, unless $g_k=0$.
\end{proof}

Note that this implies that \eqref{nonorth} is
satisfied for any QN method using an update scheme that generates
search directions that are parallel to those of CG. Also, assuming $g_k
\neq 0$, it implies that the fraction $\frac{1}{p_{k}^TB_{k}p_{k}}$ is well-defined.

\subsection{Update matrices defined by Proposition
  \ref{U_cg_2}}\label{sec:U matrix}

Having stated precise conditions on $U_k$ in Proposition \ref{U_cg_2}
we now turn to look at what these conditions imply
in terms of actual update matrices. In Theorem \ref{thmbroyden}, we show
that the conditions on $U_k$ in Proposition \ref{U_cg_2} are
equivalent to the matrix $U_k$ belonging to the one-parameter
Broyden family, \eqref{broyden2}. This implies that, under the
sufficient condition \eqref{conj}, there are no other update matrices,
even of higher rank, that make $p_k  \ / \! / \  p_k^{CG}$.

The if-direction of Theorem \ref{thmbroyden}, the fact that the one-parameter
Broyden family satisfies the conditions of Proposition \ref{U_cg_2} is
straightforward. However, the only if-directions
shows that there are no update matrices outside this family that satisfy the conditions of Proposition \ref{U_cg_2}.

\begin{theorem}\label{thmbroyden}
Assume $g_k \neq 0$. A matrix $U_k$ satisfies (i)-(iii) of Proposition \ref{U_cg_2}  if and only if $U_k$
can be expressed according to \eqref{broyden2}, the one-parameter
Broyden family, for some $\phi_k$.
\end{theorem}

\begin{proof}
Note that since $\mathcal{K}_{k-1}(p_0,H)=span\{p_0,
\dots, p_{k-2}\}$, condition (ii) of
Proposition \ref{U_cg_2} can be stated as
$$
\mathcal{K}_{k-1}(p_0,H) \subseteq \mathcal{N}(U_k),
$$
where $\mathcal{N}(U_k)=\{x\ : \ U_kx=0\}$, the \emph{null-space} of $U_k$. This implies that $\dim(\mathcal{N}(U_k)) \geq k-1$. Since $U_k$
is symmetric and applying condition (i) of
Proposition \ref{U_cg_2} it follows that
$$
\mathcal{R}(U_k) \subseteq \mathcal{K}_{k-1}(p_0,H)^{\perp} \cap
\mathcal{K}_{k+1}(p_0,H)=span\{g_{k-1}, g_k\},
$$
and that $\dim(\mathcal{R}(U_k^T))=\dim(\mathcal{R}(U_k)) \leq 2$.

Hence, one may write a
general $U_k$ that satisfies the conditions (i) and (ii) of Proposition \ref{U_cg_2} as
\begin{equation}\label{broyden3}
U_k=\left(
\begin{array}{ll}
g_{k-1} & g_k
\end{array} \right)
\left(
\begin{array}{cc}
m_{1,1} & m_{1,2}\\
m_{1,2} & m_{2,2}
\end{array} \right)
\left(
\begin{array}{c}
g_{k-1}^T\\
g_k^T
\end{array} \right).
\end{equation}

Note that due to the linear relationship between $\{
B_{k-1}p_{k-1}, Hp_{k-1}\}$  and $\{g_{k-1}, g_k\}$ it holds that
\begin{equation*}
g_{k-1}=-B_{k-1}p_{k-1}, \quad g_k=\alpha_{k-1}Hp_{k-1}+g_{k-1}.
\end{equation*}
Hence, since $\alpha_{k-1}=(p_{k-1}^T B_{k-1} p_{k-1})/(p_{k-1}^T H
p_{k-1})$ by the exact linesearch, dropping all $'k-1'$-subscripts and
letting $g_k$ be represented by $g_+$, we obtain
\begin{equation}\label{rel1}
\left(
\begin{array}{c}
g^T \\
g_+^T
\end{array} \right)
= p^TBp\left(
\begin{array}{cc}
0 & \ -1\\
1 & \ -1
\end{array} \right)
\left(
\begin{array}{c}
\frac{1}{p^T Hp}p^TH\\
\frac{1}{p^T Bp}p^TB
\end{array} \right).
\end{equation}
We may therefore rewrite \eqref{broyden3} as
\begin{equation}\label{broyden4}
U_k=\left(
\begin{array}{ll}
\frac{1}{p^T Hp}Hp & \frac{1}{p^T Bp}Bp
\end{array} \right)
\left(
\begin{array}{cc}
\hat{m}_{1,1} & \hat{m}_{1,2}\\
\hat{m}_{1,2} & \hat{m}_{2,2}
\end{array} \right)
\left(
\begin{array}{c}
\frac{1}{p^T Hp}p^TH\\
\frac{1}{p^T Bp}p^TB
\end{array} \right).
\end{equation}

Now imposing condition (iii) of Proposition \ref{U_cg_2},
\eqref{broyden4} yields
\begin{equation}\label{relUp}
U_kp=\left(
\begin{array}{ll}
\frac{1}{p^T Hp}Hp & \frac{1}{p^T Bp}Bp
\end{array} \right)
\left(
\begin{array}{cc}
\hat{m}_{1,1} & \hat{m}_{1,2}\\
\hat{m}_{1,2} & \hat{m}_{2,2}
\end{array} \right)
\left(
\begin{array}{c}
1\\
1
\end{array} \right)=Hp-Bp.
\end{equation}
A combination of Proposition~\ref{Bdefined} and \eqref{rel1} shows that
$Hp$ and $Bp$ are linearly independent. Hence, \eqref{relUp} implies
that
$$
\hat{m}_{1,1}+\hat{m}_{1,2}=p^THp , \quad \hat{m}_{1,2}+\hat{m}_{2,2}=-p^TBp.
$$
With $\hat{m}_{1,2}=-\varphi$, then
$$
\hat{m}_{1,1}=p^THp+\varphi , \quad \hat{m}_{2,2}=-p^TBp+\varphi.
$$
Substituting into \eqref{broyden4}, we obtain \eqref{broyden2}
with the scaling $\varphi=\phi_k p^TBp$. This completes the proof.
\end{proof}

The precise conditions of Proposition \ref{U_cg_2} are equivalent to choosing
$U_k$ from the one-parameter Broyden family. As seen in the proof, the
conditions (i)-(iii) of Proposition \ref{U_cg_2} may be expressed as
\begin{itemize}
\item[(a)]$\mathcal{R}(U_k) \subseteq  span\{g_{k-1}, g_k\}=\mathcal{K}_{k-1}(p_0,H)^{\perp} \cap
\mathcal{K}_{k+1}(p_0,H)$, and,
\item[(b)]$U_kp_{k-1}=(H-B_{k-1})p_{k-1}$.
\end{itemize}

Note that after we impose (a) we have three degrees of
freedom, $\hat{m}_{1,1}$, $\hat{m}_{1,2}$ and $\hat{m}_{2,2}$. Then after
imposing (b), the \emph{quasi-Newton condition}, we are left with one
degree of freedom $\phi_k$.

What is often mentioned as a feature of
the one-parameter Broyden family, using information only from the
current and previous iteration when forming $U_k$, is
in fact a condition that guarantees the equivalence to CG.  Also, the fact
that the update matrices in \eqref{broyden2} are of rank at most two is a
consequence of satisfying this condition. This is what distinguishes the one-parameter
Broyden family of updates from any $B_k$ satisfying
\eqref{conj}.

We stress that the fact that CG and QN, using a well-defined update
matrix from the one-parameter Broyden family, generates parallel
search directions and hence identical iterates is well-known. By
Proposition~\ref{U_cg_2} and Theorem~\ref{thmbroyden} we may draw the additional
conclusion that, under the
sufficient condition \eqref{conj}, there are no other update matrices,
even of higher rank, that make $p_k  \ / \! / \  p_k^{CG}$.

\subsection{Relation between $\delta_k$ and
  $\phi_k$}\label{sec:relation}
Next we derive a relation between the free parameter $\phi_k$, the Broyden
parameter used in $U_k$ when forming $B_k$ according to \eqref{Bupdate}, and the non-zero
parameter $\delta_k$ in $p_k=\delta_k p_k^{CG}$. In the following lemma we state an alternative way to express $U_k$
belonging to the one-parameter Broyden family \eqref{broyden2}.

\begin{lemma}\label{U_g}
An update matrix $U_k$ from the one-parameter Broyden family
\eqref{broyden2} can be expressed on the form
\begin{equation}\label{broyden5}
U_k=\left(
\begin{array}{ll}
g_{k-1} & g_k
\end{array} \right)
\left(
\begin{array}{cc}
\frac{p^THp}{(p^TBp)^2} -\frac{1}{p^TBp}& -\frac{p^THp}{(p^TBp)^2}\\
-\frac{p^THp}{(p^TBp)^2}& \frac{p^THp}{(p^TBp)^2}+\frac{\phi_k}{p^TBp}
\end{array} \right)
\left(
\begin{array}{c}
g_{k-1}^T\\
g_k^T
\end{array} \right),
\end{equation}
where all omitted subscripts
are $'k-1'$.
\end{lemma}
\begin{proof}
Reversing relation \eqref{rel1} gives
\begin{equation}\label{rel2}
\left(
\begin{array}{c}
\frac{1}{p^T Hp}p^TH \\
\frac{1}{p^T Bp}p^TB
\end{array} \right)
= \frac{1}{p^TBp}\left(
\begin{array}{cc}
-1 & \ 1\\
-1 & \ 0
\end{array} \right)
\left(
\begin{array}{c}
g_{k-1}^T\\
g_k^T
\end{array} \right).
\end{equation}
If \eqref{rel2} is inserted in \eqref{broyden2} with
$\varphi=\phi_k p^TBp$, then \eqref{broyden5} is obtained.
\end{proof}

In the following
proposition we show the one-to-one correspondence between the two
parameters $\phi_k$ and $\delta_k$,

\begin{proposition}\label{phi delta}
Let $B_0=I$, so that $p_0=p_0^{CG}=-g_0$. Assume that $p_i  \ / \! / \
p_i^{CG}$, for all $i \leq k-1$, that $B_{k-1}$ satisfies condition
(i) of Lemma \ref{B_cg}, that a solution $p_k$ is obtained from
\eqref{calp}, that $B_k$ is obtained from \eqref{Uconst} and that
$U_k$ belongs to the one-parameter Broyden family, \eqref{broyden2},
for some $\phi_k$, where $\phi_k \neq -(p_{k-1}^TB_{k-1}p_{k-1})/(g_k^Tg_k)$.

Then the non-zero parameter $\delta_k$ in $p_k=\delta_k p_k^{CG}$ is
given by
\begin{equation}\label{deltabroyden}
\delta_k(\phi_k)=\frac{1}{1+\phi_k\frac{g_k^Tg_k}{p_{k-1}^TB_{k-1}p_{k-1}}}.
\end{equation}
\end{proposition}

\begin{proof}
Omitting all $'k-1'$-subscripts throughout the proof. Our assumption is that $p_k$ is given by
$$
p_k=-g_k-V_{k-1}p_k -U_kp_k.
$$
By Theorem \ref{thmbroyden} $p_k  \ / \! / \
p_k^{CG}$ hence,
$$
\delta_k p_k^{CG}=-g_k-V_{k-1}\delta_k p_k^{CG} -U_k \delta_k p_k^{CG}.
$$
If the above expression is projected onto $g_k$ it holds that
$$
g_k^T(\delta_k p_k^{CG})=-g_k^Tg_k-\delta_k g_k^TV_{k-1} p_k^{CG} - \delta_k g_k^TU_k p_k^{CG}=
-g_k^Tg_k-0- \delta_k g_k^TU_k p_k^{CG},
$$
since by condition (i) of Lemma \ref{B_cg}, $\mathcal{R}(V_{k-1})
\subseteq \mathcal{K}_{k}(p_0,H)=span\{g_0, g_1, \dots, g_{k-1}\}$.
Using \eqref{broyden5} for $U_k$ it follows that
\begin{equation*}
\delta_k g_k^T p_k^{CG}=-g_k^Tg_k- \delta_k g_k^T g_k
\left(
\begin{array}{cc}
-\frac{p^THp}{(p^TBp)^2} & \quad \frac{p^THp}{(p^TBp)^2}+\frac{\phi_k}{p^TBp}
\end{array} \right)
\left(
\begin{array}{c}
g_{k-1}^T p_k^{CG}\\
g_k^T p_k^{CG}
\end{array} \right).
\end{equation*}

From \eqref{cgdir2} it follows that $g_i^Tp_k^{CG}=-g_k^Tg_k$, for all $i \leq k$, which implies that
\begin{equation*}
-\delta_k g_k^Tg_k=-g_k^Tg_k- \delta_k g_k^T g_k
\left(
\begin{array}{cc}
-\frac{p^THp}{(p^TBp)^2} & \quad \frac{p^THp}{(p^TBp)^2}+\frac{\phi_k}{p^TBp}
\end{array} \right)
\left(
\begin{array}{c}
-g_k^Tg_k\\
-g_k^Tg_k
\end{array} \right),
\end{equation*}
dividing on both sides with $-g_k^Tg_k$ yields
\begin{equation*}
\delta_k =1- \delta_k g_k^T g_k
\big(
-\frac{p^THp}{(p^TBp)^2} + \frac{p^THp}{(p^TBp)^2}+\frac{\phi_k}{p^TBp}
\big)=1- \delta_k g_k^T g_k\frac{\phi_k}{p^TBp},
\end{equation*}
which implies \eqref{deltabroyden}.

\end{proof}
Note that this result implies that for $\phi_k=0$ we get $\delta_k=1$,
i.e. $p_k^{BFGS}=p_k^{CG}$ as long as $\phi_i$, for all $i \leq k-1$,
are well-defined\footnote{See the next paragraph and Section \ref{sec:broyden defined}.}. In \cite{nazareth}, Nazareth
derives the relation between $p_k^{BFGS}$ and $p_k^{CG}$ using
induction.

Note that substituting the limit $\phi_k \to -(p_{k-1}^TB_{k-1}p_{k-1})/(g_k^Tg_k)$ into
\eqref{deltabroyden} for $g_k\ne \nolinebreak 0$ yields $\delta_k \to \infty$.
This limit of $\phi_k$ is called a \emph{degenerate value}
since it makes $B_k$
singular, which is a contradiction to our assumption that
$B_kp_k=-g_k$ is compatible. For all other values of $\phi_k$, we get a one-to-one
correspondence between $\phi_k$ and $\delta_k$ and this implies that our assumption that
$B_kp_k=-g_k$ is compatible actually implies that the solution $p_k$
is unique.

The assumption of Proposition \ref{phi delta}, that $B_{k-1}$ is chosen to satisfy condition (i) of Lemma \ref{B_cg},
will be satisfied if, in each iteration $k$, the update matrix $U_k$
belongs to the one-parameter Broyden family, \eqref{broyden2}, for
some $\phi_k$.

\subsection{Remarks on when the one-parameter
  Broyden family is well-defined}\label{sec:broyden defined}

There are several ways in which an update scheme using the
one-parameter Broyden family may become not well-defined. We have
already mentioned the degenerate value that makes $B_k$ singular in
the previous section.

In addition to the above, the matrix $U_k$ may itself become not well-defined.
For all well-defined values of $\phi_k$ it holds $U_k$ given by \eqref{broyden2}
is not well-defined if and only if $p_{k-1}^TB_{k-1}p_{k-1}=0$ and $B_{k-1}p_{k-1} \neq 0$.
It is clear that requiring $B_{k-1}$ to be definite (positive or negative) is sufficient
to avoid $U_k$ being not well-defined. However, from Proposition
\ref{Bdefined} it follows that, on (QP), $p_{k-1}^TB_{k-1}p_{k-1}=0$ and $B_{k-1}p_{k-1} \neq 0$
does not occur for any update scheme which generates search
directions parallel to those of CG.\footnote{For general functions we
  may have $p_{k-1}^TB_{k-1}p_{k-1}=0$ and $B_{k-1}p_{k-1} \neq 0$, and in \cite{fletchersinclair} the
  values of $\phi_k$ that give rise to this situation are characterized
and also termed degenerate values.}

Hence, on \eqref{qp}, in order for $U_k$, given by \eqref{broyden2}, to become not
well-defined, the undefinedness must enter in the Broyden parameter
$\phi_k$. It is well-known that the symmetric rank-one update, SR1, may
become not well-defined on (QP), see, e.g. \cite[Chapter
9]{luenberger}. SR1 is uniquely determined by
\eqref{conjqn}\footnote{An illustration that the conditions (i)-(iii)
  of Proposition \ref{U_cg_2} are indeed
  only sufficient.} and the Broyden parameter for SR1 is given by $\phi^{SR1}_k=(p_{k-1}^THp_{k-1})/(p_{k-1}^T(H-B_{k-1})p_{k-1})$,
an expression that becomes undefined for $p_{k-1}^T(H-B_{k-1})p_{k-1}=0$. Note that
this is equivalent to $\alpha_{k-1}=(p_{k-1}^TB_{k-1}p_{k-1})/(p_{k-1}^THp_{k-1})=1$.

Hence, we may
summarize our remarks on when $U_k$ given by the one-parameter Broyden
family, \eqref{broyden2}, is well-defined in the following corollary.

\begin{corollary}
Unless $\phi_k=\phi^{SR1}_k$ in \eqref{broyden2} with the unit
steplength taken in the same iteration, then $U_k$ defined by \eqref{broyden2}, the one-parameter
Broyden family, is always well-defined on \eqref{qp}.
\end{corollary}

Hence, taking the unit steplength is an indication that one needs to choose a
different update scheme than SR1 when forming $B_{k}$. We therefore
stress that hereditary (positive or negative) definiteness is not a
necessary property for the update matrices $U_k$ to be well-defined
when solving \eqref{qp}.

\section{Conclusions}\label{sec:conclusion}

The main result of this paper are the precise conditions on the update
matrix $U_k$ stated in Proposition \ref{U_cg_2}. In addition, from
Theorem \ref{thmbroyden}, we draw the conclusion that, in the
framework where we use the sufficient condition \eqref{conj} to
guarantee conjugacy of the search directions, the update schemes in QN
that give parallel search directions to those of CG are completely
described by the one-parameter Broyden family. Hence, we are able to
extend the well-known connection between CG and QN, e.g., given in
\cite{fletcherpractical}. We show that, under the sufficient condition
\eqref{conj}, there are no other update matrices, even of higher rank,
that make $p_k  \ / \! / \  p_k^{CG}$.

It seems as it may be the sufficient requirement to
get conjugate directions, \eqref{conj} that limits the freedom when choosing $U_k$ to the one-parameter Broyden
family of updates as shown by the above results. Since
this condition is only sufficient, one may still pose the
question if there are other update schemes for QN that yield the same sequence of
iterates as CG on \eqref{qp}. We believe that in order to understand
this possible limitation it will be necessary to obtain a deeper
understanding of CG.

We derive a one-to-one correspondence between the Broyden parameter
$\phi_k$ and the scaling $\delta_k$. This relation implies that the
assumption that \eqref{calp} is compatible implies uniqueness of
the solution $p_k$. If the degenerate value is used $B_k$ becomes singular.

We are also able to make the remark that the update matrices belonging
to the one-parameter Broyden
family is always well-defined on \eqref{qp}, unless the
steplength is of unit length and in the same iteration the rank-one
update is used. In this case it is the Broyden parameter $\phi_k$ that
becomes undefined.

In this paper we have focused on quadratic programming. Besides being
important in its own right, it is also a highly important as a subproblem when
solving unconstrained optimization problems. For a survey on methods
for unconstrained optimization see, e.g., \cite{recentadv}. Also, it
deserves mentioning that work has been done on QN
update schemes for general unconstrained optimization
considering \eqref{conj} for only $j=k-1$ and $j=k-2$, deriving an
update scheme that satisfies the quasi-Newton condition and has a
minimum violation of it for the previous step, see
\cite{mifflinnazareth, nazareth2}.

A further motivation for this research in this paper is that the deeper understanding
of what is important in the choice of $U_k$ could be implemented in a
limited\nolinebreak-memory QN method. I.e., can one choose which columns to save
based on some other criteria than just picking the most recent ones? See, e.g., \cite[Chapter 9]{nocedalwright}, for an introduction to limited\nolinebreak-memory QN methods.

Finally, it should be pointed out that the discussion of this paper is
limited to exact arithmetic. Even in cases where CG and QN generate
identical iterates in exact arithmetic, the difference between
numerically computed iterates by the two methods may be quite large,
see, e.g.~\cite{hager}.

{\small
\subsubsection*{Acknowledgements:}
We thank the referees and the associate editor for their insightful
comments and suggestions which significantly improved the presentation
of this paper.
}


Monotonicity of solutions for some nonlocal elliptic problems in half-spaces

Abstract: In this paper we consider classical solutions $u$ of the semilinear fractional problem $(-\Delta)^s u = f(u)$ in $\mathbb{R}^N_+$ with $u=0$ in $\mathbb{R}^N \setminus \mathbb{R}^N_+$, where $(-\Delta)^s$, $0<s<1$, stands for the fractional laplacian, $N\ge 2$, $\mathbb{R}^N_+=\{x=(x',x_N)\in \mathbb{R}^N:\ x_N>0\}$ is the half-space and $f\in C^1$ is a given function. With no additional restriction on the function $f$, we show that bounded, nonnegative, nontrivial classical solutions are indeed positive in $\mathbb{R}^N_+$ and verify $$ \frac{\partial u}{\partial x_N}>0 \quad \hbox{in } \mathbb{R}^N_+. $$ This is in contrast with previously known results for the local case $s=1$, where nonnegative solutions which are not positive do exist and the monotonicity property above is not known to hold in general even for positive solutions when $f(0)<0$.

\section{Introduction}
\setcounter{section}{1}
\setcounter{equation}{0}

The objective of the present paper is to deal with the semilinear problem
\begin{equation}\label{problema}
\left\{
\begin{array}{ll}
(-\Delta)^s u = f(u) & \hbox{in }\mathbb R^N_+,\\[0.35pc]
\ \ u=0 & \hbox{in }\mathbb R^N \setminus \mathbb R^N_+,
\end{array}
\right.
\end{equation}
where $N\ge 2$, $\mathbb R^N_+=\{x=(x',x_N)\in \mathbb R^N:\ x_N>0\}$ is the half-space and $f$ is a $C^1$ function.
The operator $(-\Delta)^s$, $0<s<1$, is the well-known \emph{fractional laplacian}, which
is defined on smooth functions as
\begin{equation}\label{operador}
(-\Delta)^s u(x) =  \int_{\mathbb R^N} \frac{u(x)-u(y)}{|x-y|^{N+2s}} dy,
\end{equation}
up to a normalization constant which will be omitted for brevity. The integral in \eqref{operador}
has to be understood in the principal value sense, that is, as the limit as $\varepsilon\to 0$ of the
same integral taken in the complement of the ball $B_\varepsilon(x)$ of center $x$ and radius $\varepsilon$. Alternatively, this
operator can be defined (omitting again the normalization constant) as
\begin{equation}\label{operador2}
(-\Delta)^s u(x) =\frac{1}{2} \int_{\mathbb R^N} \frac{2u(x) -u(x+y)-u(x-y)}{|y|^{N+2s}} dy,
\end{equation}
where now the integral is absolutely convergent for sufficiently smooth functions.

\smallskip
\smallskip

Problems with nonlocal diffusion related to \eqref{problema} have been intensively
studied in the last years, after their appearance when modeling different situations.
For instance, anomalous diffusion and quasi-geostrophic flows, turbulence and
water waves, molecular dynamics and relativistic quantum mechanics of stars
(see \cite{BoG,CaV,Co,TZ} and references); or mathematical
finance (cf. \cite{A,Be,CoT}), elasticity problems \cite{signorini},
thin obstacle problem \cite{Caf79}, phase transition \cite{AB98, CSM05, SV08b},
crystal dislocation \cite{dfv, toland} and stratified materials \cite{savin_vald}.

\medskip

Our inspiration to study problem \eqref{problema} comes from the local case
$s=1$, that is
\begin{equation}\label{problema-local}
\left\{
\begin{array}{ll}
-\Delta u = f(u) & \hbox{in }\mathbb R^N_+,\\[0.35pc]
\ \ u=0 & \hbox{on } \partial \mathbb R^N_+.
\end{array}
\right.
\end{equation}
In a seminal series of papers (cf. \cite{BCN1, BCN2, BCN3, BCN4}), Berestycki, Caffarelli and
Nirenberg obtained interesting
qualitative properties for positive solutions of \eqref{problema-local} and Lipschitz nonlinearities
$f$. Among other results, they showed that if $f(0)\ge 0$, then
any positive solution of \eqref{problema-local} verifies
\begin{equation}\label{eq-monotonia-local}
\frac{\partial u}{\partial x_N}>0 \quad \hbox{in }\mathbb R^N_+
\end{equation}
(see \cite{BCN2} or \cite{BCN3}). This property had been shown initially with additional
assumptions on both the solutions and the nonlinearities by Dancer in \cite{D1} and \cite{D2}.

The case $f(0)< 0$ is, however, more subtle, and only partial results are known for the
moment. See \cite{BCN3}, \cite{FS} for the case $N=2$, \cite{FSo} for $N=2,3$ and
\cite{CEGM} for dimensions $N\ge 2$. The main reason for this difference is the existence of
nonnegative (periodic) solutions which are not strictly positive.

\medskip

With regard to a similar property as \eqref{eq-monotonia-local} for solutions of the nonlocal
problem \eqref{problema}, only some partial results have been achieved so far, at the best of
our knowledge. Let us mention \cite{FW} and \cite{QX} where monotone, positive
nonlinearities where considered, and \cite{chinos} for the particular case $f(t)=t^p$, $p>1$.
On the other hand, the very recent preprint \cite{DSV} analyzes the same question in more
general domains, but with a very restricted class of nonlinearities.

Our intention is to show that actually property \eqref{eq-monotonia-local} continues to be
true for bounded, nonnegative, nontrivial solutions of \eqref{problema} \emph{with no additional
assumptions placed on the nonlinearity} $f$ aside its regularity. This is in striking contrast
with problem \eqref{problema-local}, where, as we have remarked, the case $f(0)<0$
remains unsolved in its full generality for the moment.

\medskip

Throughout this work, we will deal with bounded, classical solutions of \eqref{problema}.
However, this will not cause a loss in generality, since it is well-known from the regularity
theory developed in  \cite{S, CS, CS2} and bootstrapping that bounded, viscosity solutions
of \eqref{problema} are automatically classical. Observe that classical
solutions verify $u\in C^{2s+\beta}(\mathbb R^N_+)$ for every $\beta\in (0,1)$, and in particular they are
seen to be in $C^1(\mathbb R^N_+)$.

\medskip

Our main result is the following:

\begin{teorema}\label{teorema-1}
Assume $f\in C^1(\mathbb R)$ and let $u$ be a bounded, nonnegative, nontrivial classical solution of \eqref{problema}.
Then $u$ is positive and
\begin{equation}\label{eq-derivadapos}
\frac{\partial u}{\partial x_N}>0 \quad \hbox{in } \mathbb R^N_+.
\end{equation}
\end{teorema}

\bigskip

As a consequence of Theorem \ref{teorema-1}, we can also obtain some Liouville theorems for
problem \eqref{problema} with some special nonlinearities.

\begin{teorema}\label{teorema-2}
Assume $f\in C^1(\mathbb R)$ is such that $f'(t)>0$ for $t>0$, and one of the following holds:

\begin{itemize}

\item[(a)] $f(0)\ne 0$;

\smallskip

\item[(b)] $f(0)=0$ and $f'(0)>0$.

\end{itemize}
Then problem \eqref{problema} does not admit bounded, nonnegative, nontrivial
solutions.
\end{teorema}

\bigskip

An interesting particular case in Theorem \ref{teorema-2} is obtained when we set
$f(t)=t-1$. In this case the differences between the local version \eqref{problema-local} and
its nonlocal counterpart \eqref{problema} become more evident, since in the former
there exists a unique nonnegative solution given by $u(x)=1-\cos x_N$ (see \cite{CEGM}),
while for the latter we have:

\begin{coro}\label{coro-lineal}
The problem
$$
\left\{
\begin{array}{ll}
(-\Delta)^s u = u-1 & \hbox{in }\mathbb R^N_+,\\[0.35pc]
\ \ u=0 & \hbox{in }\mathbb R^N \setminus \mathbb R^N_+
\end{array}
\right.
$$
does not admit any bounded, nonnegative, nontrivial solution.
\end{coro}

\medskip

It is interesting to remark that Theorem \ref{teorema-1} is an important tool to prove other
Liouville theorems for bounded solutions of \eqref{problema}. Indeed, passing to the
limit as $x_N\to +\infty$, we find that such solutions converge to a \emph{stable} solution
of $(-\Delta)^s u=f(u)$ in $\mathbb R^{N-1}$. Then one can use the nonexistence theorems already
available in that situation (cf. for instance \cite{DuS}).

\medskip

We conclude the introduction with a couple of comments on our proofs. We use
moving planes to show that any nonnegative, bounded, classical solution of \eqref{problema}
is monotone in the $x_N$ direction. To deal with the moving planes method, we mainly follow
the approach in \cite{FW}. However, instead of representing the solutions of \eqref{problema}
with the aid of Green's function in $\mathbb R^N_+$ at the onset, we use it for an adequate
truncation related to $u$ and its reflections. This allows us to avoid any monotonicity or
sign restriction on $f$.

It is to be noted that at one point in the argument, when it is assumed that the moving of
the planes stop somewhere, we need to rule out the existence of solutions which are
symmetric with respect to a hyperplane contained in $\mathbb R^N_+$. In the local case, this is
only possible under the additional restriction $f(0)\ge 0$, since such solutions do exist
if $f(0)<0$. However, we show in the present work that symmetric solutions can not
exist with no additional restriction on $f$ (see
Theorem \ref{lema-noexistencia} below). In our opinion, this is a result of independent interest.
Its proof is based on the regularity inherited by symmetry in the strip, which allows to evaluate
the equation on the boundary of the half space. Then the nonlocality of the operator implies,
loosely speaking, that the interactions between points which are far away in $\mathbb R^N_+$ is too strong
and the solution must vanish. This is a remarkable difference with respect to the case $s=1$,
where this interaction is not present.

\medskip

The rest of the paper is organized as follows: in Section 2 we give some preliminaries
and introduce the notation to be used for the moving planes. Section 3 deals with some
properties of the Green's function in a half-space taken from \cite{FW} and with a
different representation in terms of this function. In Section 4, we obtain a nonexistence
result for bounded, nonnegative, nontrivial solutions which are symmetric with respect to
a hyperplane, and in Section 5 we perform the proof of our main results, Theorems
\ref{teorema-1} and \ref{teorema-2}.

\bigskip

\section{Some preliminaries}\label{s2}
\setcounter{section}{2}
\setcounter{equation}{0}

In this section, we gather some preliminary properties which will be useful in the
forthcoming sections. We notice that, although we are mostly concerned with solutions in
the classical sense, other more general concepts of solutions have to be
considered at some places in the present work, mainly due to the fact that we work with truncations
of the original functions.

Thus, throughout this section we will consider inequalities in the viscosity sense (see \cite{CS}
for a definition). We begin by considering a version of the maximum principle for
the operator $(-\Delta)^s$ in unbounded domains, which will be needed below.
We believe that this result is new.

\begin{lema}\label{lema-pmax}
Assume $D\subset \mathbb R^N_+$ is a domain and let $u\in C(\mathbb R^N)$ be a bounded function verifying
$(-\Delta)^s u\ge 0$ in $D$ in the viscosity sense, with $u\ge 0$ in $\mathbb R^N \setminus D$. Then
$u\ge 0$ in $D$.
\end{lema}

\begin{proof}
First of all observe that the function $\varphi(x)=  (x_N)_+^s$ is $s-$harmonic in $\mathbb R^N_+$,
where $(x_N)_+$ is the function which coincides with $x_N$ in $(0,+\infty)$ and vanishes in
$(-\infty,0]$. Indeed, when $x\in \mathbb R^N_+$:
\begin{align*}
(-\Delta)^{s}\varphi (x) & =\int_{\mathbb{R}}{\frac{(x_N)^{s}-(y_N)_+^{s}}{|x_N-y_N|^{1+2s}}
\hspace{-.5mm} \left(\int_{\mathbb{R}^{N-1}} \hspace{-1mm}
{\frac{|x_N-y_N|^{1+2s}}{\left((x_N-y_N)^2+|x'-y'|^2\right)^{\frac{N+2s}{2}}}\, dy'} \hspace{-1mm}
\right) \hspace{-1mm} dy_N}\\
&=\int_{\mathbb{R}}{\frac{(x_N)^{s}-(y_N)_+^{s}}{|x_N-y_N|^{1+2s}} \hspace{-.5mm} \left(\int_{\mathbb{R}^{N-1}}
\frac{dz}{(1+z^2)^{\frac{N+2s}{2}}}\right)dy_N}\\
&=C \int_{\mathbb{R}} \frac{(x_N)^{s}-(y_N)_+^{s}}{|x_N-y_N|^{1+2s}} dy_N=0
\end{align*}
(see for instance the introduction in \cite{CJS} or Proposition 3.1 in \cite{ROS}).

Next take $\varepsilon>0$ and consider the function
$$
v_\varepsilon(x)= u(x) + \varepsilon (x_N)_+^s, \ x\in \mathbb R^N_+.
$$
Since $u$ is bounded, there exists $M>0$ such that $v_\varepsilon \ge 0$ if $x_N\ge M$. Define the set
$D_M=D \cap \{x\in \mathbb R^N:\ 0<x_N<M\}$. Then, in the viscosity sense,
\begin{equation}\label{later}
\left\{
\begin{array}{lll}
(-\Delta)^s v_\varepsilon \geq 0 & \hbox{in } D_M,\\[0.35pc]
\ \ v_\varepsilon  \ge 0 & \hbox{in }\mathbb R^N \setminus D_M.
\end{array}
\right.
\end{equation}
We are in a position to apply Theorem 2.3 in \cite{QX} to conclude that $v_\varepsilon \ge 0$ in $\mathbb R^N$. Letting
$\varepsilon\to 0$, we obtain that $u\ge 0$ in $\mathbb R^N$. It is worth remarking that, although Theorem 2.3 in
\cite{QX} is stated for functions which vanish in $\mathbb R^N\setminus D_M$, a careful inspection
shows that it is still valid when the involved functions are nonnegative there.
\end{proof}

\bigskip

Before giving our next result, let us introduce some notation related to the method
of moving planes. For $\lambda>0$ we denote, as customary:
$$
\begin{array}{l}
\Sigma_\lambda :=\{ x\in \mathbb R^N_+:\ 0<x_N<\lambda\}\\[.5pc]
T_\lambda :=\{x\in \mathbb R^N:\ x_N=\lambda\}\\[.5pc]
x^\lambda:=(x',2\lambda-x_N) \ \hbox{(the reflection of }x \hbox{ with respect to } T_\lambda).
\end{array}
$$
If $f$ is a given nonlinearity and $u$ stands for a a bounded, classical nonnegative
solution of our problem \eqref{problema} we also set
$$
\begin{array}{ll}
u_\lambda(x)= \left\{
\begin{array}{ll}
u(x), & x\in \Sigma_\lambda \cup (\mathbb R^N \setminus \mathbb R^N_+)\\
u(x^\lambda), & x\not\in \Sigma_\lambda \cup (\mathbb R^N \setminus \mathbb R^N_+)
\end{array}
\right. \\[0.75pc]
w_\lambda(x)= u_\lambda(x)-u(x), \quad x\in \mathbb R^N.
\end{array}
$$
Since our ultimate objective is to show that $w_\lambda$ is always nonnegative in $\Sigma_\lambda$, the
following will also be relevant:
$$
\begin{array}{l}
D_\lambda=\{x\in \Sigma_\lambda: \ w_\lambda(x)<0\}\\[0.5pc]
W_\lambda=\{x\in D_\lambda: \ f(u(x)) > f(u^\lambda(x))\}\\[0.5pc]
v_\lambda = w_\lambda \chi_{D_\lambda},
\end{array}
$$
where $\chi$ will stand throughout the paper for the characteristic function of a set.
It is plain that the function $v_\lambda$ will only be meaningful when $w_\lambda$ is negative somewhere
in $\Sigma_\lambda$. We next state one of its important properties.

\begin{lema}\label{lema-signo}
Assume $w_\lambda<0$ somewhere in $\Sigma_\lambda$, for some $\lambda>0$. Then,
\begin{equation}\label{desigualdad}
(-\Delta)^s v_\lambda \ge (f(u^\lambda)-f(u)) \chi_{D_\lambda} \quad \hbox{in } \mathbb R^N_+,
\end{equation}
in the viscosity sense.
\end{lema}

\begin{proof}
Let us prove first that, when $x\in D_\lambda$, \eqref{desigualdad} holds in the classical
sense (cf. the proof of Theorem 1.1 in \cite{FeWa}). Denote
$$
z_\lambda = w_\lambda-v_\lambda.
$$
It is clear that in $D_\lambda$  \eqref{desigualdad} is equivalent to $(-\Delta)^s z_\lambda\le 0$. To prove this last
inequality, denote by $E_\lambda$ the reflection through the hyperplane $T_\lambda$ of $D_\lambda$. Using that
$z_\lambda \equiv 0$ in $D_\lambda$ and $z_\lambda \ge 0$ in $\Sigma_\lambda\setminus D_\lambda$, we have for every
$x\in D_\lambda$:
\begin{align*}
(-\Delta)^s z_\lambda (x) &=-\left(  \int_{\Sigma_\lambda \setminus D_\lambda} + \int_{E_\lambda}+ \int_{\Sigma_\lambda^c \setminus E_\lambda} \right)
\frac{z_\lambda(y)}{|x-y|^{N+2s}} dy\nonumber\\
& \leq  -\left(  \int_{\Sigma_\lambda \setminus D_\lambda} +  \int_{\Sigma_\lambda^c \setminus E_\lambda} \right)
\frac{z_\lambda(y)}{|x-y|^{N+2s}} dy \nonumber\\
& = -  \int_{\Sigma_\lambda \setminus D_\lambda} z_\lambda(y) \left( \frac{1}{|x-y|^{N+2s}} - \frac{1}{|x-y^\lambda|^{N+2s}} \right)dy
\le 0.
\end{align*}
Here we have used that $|x-y|\le |x-y^\lambda|$ when $x\in D_\lambda$, $y\in \Sigma_\lambda$, which can be
easily checked. Thus \eqref{desigualdad} is proved in $D_\lambda$.

On the other hand, when $x\in \mathbb R^N_+\setminus \overline{D_\lambda}$, it is immediate that
$$
(-\Delta)^s v_\lambda (x)=- \int_{D_\lambda} \frac{v_\lambda (y)}{|x-y|^{N+2s}} dy \ge 0,
$$
since $v_\lambda=0$ in $\mathbb R^N_+\setminus D_\lambda$ and $v_\lambda<0$ in $D_\lambda$. Therefore \eqref{desigualdad}
also holds in the classical sense in $\mathbb R^N_+\setminus \overline{D_\lambda}$.

However, the function $v_\lambda$ needs not be smooth on $\partial D_\lambda$, so that it is not to be
expected that its fractional laplacian is even well-defined there. But the inequality can be
checked in the viscosity sense. To prove this, take $x_0\in \partial D_\lambda$ and let $\varphi\in C^\infty(\mathbb R^N)$
be such that $\varphi <v_\lambda$ in a reduced neighborhood $\mathcal{U}\setminus \{x_0\}$ of
$x_0$, with $\varphi(x_0)=v_\lambda(x_0)=0$. Then $(-\Delta)^s v_\lambda (x_0)\ge 0$ means
$(-\Delta)^s \psi (x_0) \ge 0$, where
$$
\psi(x)= \left\{
\begin{array}{ll}
\varphi(x) & x\in \mathcal{U}\\
v_\lambda(x) & x\in \mathbb R^N\setminus \mathcal{U}
\end{array}
\right.
$$
(cf. \cite{CS}). The inequality $(-\Delta)^s \psi(x_0)\ge 0$ is easily checked since,
taking into account that $v_\lambda \le 0$ in $\mathbb R^N$, so that $\varphi\le 0$ in $\mathcal{U}$,
we deduce
$$
(-\Delta)^s \psi (x_0)=- \int_{\mathcal{U}} \frac{\varphi(y)}{|x-y|^{N+2s}} dy
-  \int_{\mathbb R^N\setminus \mathcal{U}} \frac{v_\lambda (y)}{|x-y|^{N+2s}} dy\ge 0,
$$
as was to be shown.
\end{proof}

\medskip

\section{A representation in the half-space}
\setcounter{section}{3}
\setcounter{equation}{0}

In this section, we will show that the function $v_\lambda$ verifies an inequality
which involves the Green's function in the half-space. As we have already remarked
in the Introduction, the representation is rather general and does not impose any
additional properties on the nonlinear term $f$. Recall that we are always assuming
$0<s<1$.

We introduce the Green's function for $\mathbb R^N_+$ (see \cite{FW}). If $x,y\in \mathbb R^N_+$, we let
\begin{equation}\label{eq-green}
G_\infty^+(x,y) = \frac{k_N^s}{2} |x-y|^{2s-N} \int_0 ^{\psi_\infty^+(x,y)} \frac{t^{s-1}}{(t+1)^\frac{N}{2}} dt,
\end{equation}
where
$$
\psi_\infty^+ (x,y) = \frac{4 x_N y_N}{|x-y|^2}.
$$
In \eqref{eq-green}, $k_N$ is a positive constant whose actual value is immaterial for us.
It is shown in Theorem 3.1 of \cite{FW} that if $u\in L^\infty(\mathbb R^N)$ vanishes outside $\mathbb R^N_+$ and
$(-\Delta)^s u\in L^\infty(\mathbb R^N_+)$ is nonnegative, then
$$
u(x) = \int_{\mathbb R^N_+} G_\infty^+ (x,y) (-\Delta)^s u(y) dy, \quad x\in \mathbb R^N_+.
$$
To avoid the sign restriction just mentioned, we follow a different approach. The information
we obtain is slightly weaker, but it suffices for our arguments in the proofs of Section 5. Here
is the main result of this section:

\begin{lema}\label{lema-varianteFW}
Assume $f$ is locally bounded and let $u$ be a nonnegative, bounded solution of \eqref{problema}.
Suppose $w_\lambda<0$ somewhere in $\Sigma_\lambda$, for some $\lambda>0$. Then, for
every $x\in W_\lambda=\{x\in D_\lambda: \ f(u(x)) > f(u^\lambda(x))\}$,
$$
v_\lambda(x) \ge \int_{W_\lambda} G_\infty^+ (x,y) (f(u^\lambda(y))-f(u(y))) dy,
$$
where the integral is absolutely convergent.
\end{lema}

\medskip

In order to prove Lemma \ref{lema-varianteFW}, we borrow some notation and results
from \cite{FW}. For $R>0$, define $B_R^+:=B_R (Re_N)\subset \mathbb R^N_+$, where $e_N$ stands for the
last vector in the canonical basis, and let
$$
G_R^+(x,y)= \frac{k_N^s}{2} |x-y|^{2s-N} \int_0 ^{\psi_R^+(x,y)} \frac{t^{s-1}}{(t+1)^\frac{N}{2}} dt,
$$
with
$$
\psi_R^+(x,y) = \frac{(R^2 - |x-R e_N|^2)(R^2 - |y-R e_N|^2)}{R^2 |x-y|^2},
$$
be the Green's function in the ball $B_R^+$. We also introduce
$$
\Gamma_R^+ (x,y)= C_{N,s} \left( \frac{R^2 - |x- Re_N|^2}{|y- Re_N|^2 - R^2}\right)^s |x-y|^{-N},
$$
the Poisson kernel for the same ball (cf. \cite{BGR} for some properties of both functions). According to
Corollary 2.9 in \cite{FW}, if $h_R$ is the unique solution of the Dirichlet problem
$$
\left\{
\begin{array}{ll}
(-\Delta)^s h = g_1 & \hbox{in } B_R^+,\\[0.35pc]
\ \ h=g_2 & \hbox{in }\mathbb R^N \setminus B_R^+,
\end{array}
\right.
$$
where $g_1\in L^\infty(B_R^+)$ and $g_2\in L^\infty(\mathbb R^N\setminus B_R^+)$, then we can write:
\begin{equation}\label{eq-repres}
h_R (x) = \int_{B_R^+} G_R^+(x,y) g_1(y) dy + \int_{\mathbb R^N \setminus B_R^+} \Gamma_R^+(x,y) g_2(y) dy.
\end{equation}
Regarding this representation, it is to be noted that, when $g_2 \in L^\infty(\mathbb R^N)$, as a consequence
of equation (3.7) in \cite{FW}, then
\begin{equation}\label{eq-limite-gamma}
\lim_{R\to +\infty} \int_{\mathbb R^N \setminus B_R^+} \Gamma_R^+(x,y) g_2(y) dy=0
\end{equation}
for every $x\in \mathbb R^N_+$.
The following properties of Green's function will be used in our proof of
Lemma \ref{lema-varianteFW} and in the proof of Theorem \ref{teorema-1} in Section 5.

\begin{lema}\label{lema-FW}
Fix $R_0>0$. Then the functions $G_R^+(x,y)$ are nondecreasing with respect to $R$ in
$B_{R_0}^+ \times B_{R_0}^+$ if $R>R_0$ and verify
$$
G_R^+ \to G_\infty^+  \quad \hbox{in }B_{R_0}^+\times B_{R_0}^+ \hbox{ as }R\to +\infty.
$$
Moreover, for every $\lambda>0$, there exists $C=C(N,s,\lambda)$ such that
\begin{equation}\label{est-green}
G_\infty^+(x,y) \le C \min\{ |x-y|^{2s-N},|x-y|^{-N}\} \quad \hbox{for } x,y \in \Sigma_\lambda.
\end{equation}
In addition, the function $G_\infty^+(x,y)$ enjoys the following properties:

\begin{itemize}

\item[(a)] If $\{x_n\}$ is a bounded sequence, then for every $\lambda>0$
$$
\lim_{R\to +\infty} \int_{\Sigma_\lambda \cap B_R^c} G_\infty^+ (x_n,y) dy =0,
$$
uniformly in $n$.

\item[(b)] If $\lambda>0$ and $\{x_n\}$ is a bounded sequence, then for every $R>0$ there exists
a positive constant $C$ such that
$$
\int_{\Sigma_\lambda \cap B_R} G_\infty^+(x_n,y) dy\le C \quad \hbox{for every } n\in \mathbb N.
$$

\item[(c)] For every $\lambda>0$ and $x_0\in \partial \mathbb R^N_+$, we have
$$
\lim_{x\to x_0} \int_{\Sigma_\lambda} G_\infty^+ (x,y) dy =0.
$$
\end{itemize}
\end{lema}

\begin{proof}[Sketch of proof]
The statements about monotonicity and convergence of $G_R^+$ are a consequence
of Lemma 3.2 in \cite{FW}. The estimates \eqref{est-green} follow because of Lemma
4.1 there.

Parts (a) and (b) are a direct consequence of \eqref{est-green}, while for the proof of (c),
we only have to notice that, if $h\in C(\mathbb R^N)$ is the unique solution of
$$
\left\{
\begin{array}{ll}
(-\Delta)^s h = \chi_{\Sigma_\lambda}  & \hbox{in }\mathbb R^N_+,\\[0.35pc]
\ \ h=0 & \hbox{in }\mathbb R^N \setminus \mathbb R^N_+,
\end{array}
\right.
$$
then by Theorem 3.1 in \cite{FW} we have
$$
h(x) =\int_{\Sigma_\lambda}G_\infty ^+(x,y) dy,
$$
and the proof follows because of the continuity of $h$ up to the boundary of
$\mathbb R^N_+$.
\end{proof}

\bigskip

We can now proceed to the proof of Lemma \ref{lema-varianteFW}.

\medskip

\begin{proof}[Proof of Lemma \ref{lema-varianteFW}]
We start by observing that, by Lemma \ref{lema-signo}
\begin{equation}\label{eq4}
(-\Delta)^s v_\lambda \ge (f(u^\lambda)-f(u)) \chi_{D_\lambda} \quad \hbox{in } \mathbb R^N_+.
\end{equation}
Consider the balls $B_R^+$ introduced before and denote by $h_R$ the unique solution
of the problem
\begin{equation}\label{prob-bola}
\left\{
\begin{array}{ll}
(-\Delta)^s h = (f(u^\lambda)-f(u)) \chi_{D_\lambda} & \hbox{in } B_R^+,\\[0.35pc]
\ \ h=v_\lambda & \hbox{in }\mathbb R^N \setminus  B_R^+.
\end{array}
\right.
\end{equation}
It is clear by \eqref{eq4} and the maximum principle that $v_\lambda \ge h_R$ in $B_R^+$. Therefore,
according to \eqref{eq-repres}, we may write
\begin{equation}\label{comp-green}
\begin{array}{rl}
v_\lambda(x)  \ge h_R(x) \hspace{-2mm} & \displaystyle = \int_{B_R^+\cap D_\lambda} G_R^+(x,y) (f(u^\lambda(y))-f(u(y))) dy \\[1pc]
&+\displaystyle \int_{\mathbb R^N_+ \setminus B_R^+} \Gamma_R^+(x,y) v_\lambda(y) dy.
\end{array}
\end{equation}
Our intention is to pass to the limit in \eqref{comp-green}. Notice that, since $v_\lambda$ is
bounded, we have by \eqref{eq-limite-gamma} that the last integral converges to zero as
$R\to +\infty$.

\smallskip

On the other hand, we obtain from Lemma \ref{lema-FW} that $G_R^+$ is nondecreasing
as a function of $R$ and, for fixed $x\in \Sigma_{\lambda}$
$$
G_R^+(x,y) \le G_\infty ^+ (x,y) \le C \min\{|x-y|^{-N+2s},|x-y|^{-N}\} \in L^1(\Sigma_{\lambda}),
$$
as a function of $y$. Therefore, letting $R\to +\infty$ in \eqref{comp-green} and using dominated
convergence we arrive at
$$
\begin{array}{rl}
v_\lambda(x) \hspace{-2mm} & \displaystyle \ge \int_{D_\lambda} G_\infty^+(x,y) (f(u^\lambda(y))-f(u(y))) dy \\[1pc]
& \ge \displaystyle \int_{W_\lambda} G_\infty^+(x,y) (f(u^\lambda(y))-f(u(y))) dy,
\end{array}
$$
as was to be proved.
\end{proof}

\medskip

\begin{obs}\label{rem-semiespacio}
{\rm Similar results as the ones given in this section follow easily in other half-spaces by means of a
simple change of variables. For instance, in $H:=\{x\in \mathbb R^N:\ x_N<\lambda\}$, the Green's
function is given by
$$
G(x;y)=G_\infty^+ (x',\lambda-x_N;y',\lambda-y_N), \quad x,y\in H,
$$
and similar properties as those given in Lemma \ref{lema-varianteFW} are obtained at once.
}
\end{obs}

\bigskip

\section{A nonexistence theorem}
\setcounter{section}{4}
\setcounter{equation}{0}

In this section we will state and prove a nonexistence result for nonnegative solutions of \eqref{problema}
which are symmetric with respect to a hyperplane. This result is fairly important in the moving
planes argument, and it is the reason why the case $f(0)<0$ can be included in our
proofs, in contrast with the local case $s=1$. We believe it is interesting in
its own right.

It is to be noted that, when the nonlinearity verifies $f(0)\ge 0$, the nonexistence
of these symmetric solutions is a direct consequence of the strong maximum principle.
The proof we give, however, covers also this case. Observe that next theorem holds with
minimal hypotheses on $f$.

\begin{teorema}\label{lema-noexistencia}
Assume $f$ is continuous at zero and let $u\in C^{2s+\beta}(\mathbb R^N_+)$ ($0<\beta<1$) be a bounded,
nonnegative, classical solution of
\eqref{problema} which is symmetric with respect to $T_\lambda$ in $\Sigma_{2\lambda}$ for some $\lambda>0$,
that is
$$
u(x',2\lambda-x_N)=u(x',x_N), \quad x\in \Sigma_{2\lambda}.
$$
Then $f(0)=0$ and $u\equiv 0$ in $\mathbb R^N$.
\end{teorema}

\begin{proof}
We begin by showing that $u$ verifies the equation at $x=0$, that is,
\begin{equation}\label{eq-claim-1}
\int_{\mathbb R^N_+} \frac{u(y)}{|y|^{N+2s}} dy = -f(0).
\end{equation}
To see this, we first observe that with no loss of generality we may assume that $\beta$ is
restricted to satisfy $2s+\beta<2$. Thus the
symmetry of $u$ implies that the same regularity holds up to the boundary of $\mathbb R^N_+$, since
necessarily $u$ and $\nabla u$ vanish there. Therefore
$u\in C^{2s+\beta}(\mathbb R^N)$.

Take an arbitrary sequence $\{x_n\} \subset \Sigma_{2\lambda}$ with $x_n\to 0$. Evaluating the
equation at $x_n$, but using expression \eqref{operador2} for $(-\Delta)^s$, we see that
\begin{equation}\label{eq1}
f(u(x_n)) =\frac{1}{2} \int_{\mathbb R^N} \frac{2u(x_n)-u(x_n+y)-u(x_n-y)}{|y|^{N+2s}}\; dy.
\end{equation}
Now we have to distinguish between the cases $0<s<\frac{1}{2}$ and $\frac{1}{2}\le s<1$.
In the former case, assuming $\beta$ is such that $2s+\beta<1$, we deduce from the regularity of $u$ that
for sufficiently large $n$:
\begin{equation}\label{holder1}
|u(x_n)-u(x_n+y)| \le C |y|^{2s+\beta} \quad \hbox{whenever } |y|\le 1,
\end{equation}
for some positive constant $C$. In the latter, if $\beta$ is such that $2s+\beta<2$, the regularity implies,
also for large enough $n$
\begin{equation}\label{holder2}
|u(x_n)-u(x_n+y)- u(x_n-y)|  \le C |y|^{2s+\beta} \quad \hbox{if } |y|\le 1.
\end{equation}
On the other hand, for $|y| \ge 1$,
\begin{equation}\label{holder3}
\left| \frac{u(x_n)-u(x_n+y)-u(x_n-y)}{|y|^{N+2s}}  \right| \le 3 \|u\|_{L^\infty (\mathbb R^N_+)} |y|^{-N+2s}.
\end{equation}
Inequalities \eqref{holder1}, \eqref{holder2} and \eqref{holder3} show that the integrand in \eqref{eq1}
is bounded in absolute value by a function which is in $L^1(\mathbb R^N)$. Therefore, we may pass to the
limit in \eqref{eq1} with the aid of dominated convergence to arrive at
$$
f(0) =- \frac{1}{2} \int_{\mathbb R^N} \frac{u(y)+u(-y)}{|y|^{N+2s}}\; dy,
$$
which is equivalent to \eqref{eq-claim-1}.

By evaluating the equation at $x=2\lambda e_N$, using the fact that $u(2\lambda e_N)=0$
by symmetry, we deduce from \eqref{eq-claim-1} that
\begin{equation}\label{eq2}
\int_{\mathbb R^N_+} \frac{u(y)}{|y|^{N+2s}} dy =\int_{\mathbb R^N_+} \frac{u(y)}{|2\lambda e_N-y|^{N+2s}} dy.
\end{equation}
Next, we split the second integral in two parts and use the symmetry of $u$ in $\Sigma_{2\lambda}$
to have:
\begin{align*}
\int_{\mathbb R^N_+} \frac{u(y)}{|2\lambda e_N-y|^{N+2s}} dy & = \left( \int_{\Sigma_{2\lambda}} +
 \int_{\mathbb R^N_+\setminus \Sigma_{2\lambda}} \right) \frac{u(y)}{|2\lambda e_N-y|^{N+2s}} dy\\[0.5pc]
& =\int_{\Sigma_{2\lambda}} \frac{u(z)}{|z|^{N+2s}} dz +
 \int_{\mathbb R^N_+\setminus \Sigma_{2\lambda}} \frac{u(y)}{|2\lambda e_N-y|^{N+2s}} dy.
\end{align*}
Hence, from \eqref{eq2} we see that
$$
\int_{\mathbb R^N_+ \setminus \Sigma_{2\lambda}} \frac{u(y)}{|y|^{N+2s}} dy = \int_{\mathbb R^N_+ \setminus \Sigma_{2\lambda}}
\frac{u(y)}{|2\lambda e_N-y|^{N+2s}} dy.
$$
Taking into account that $u\ge 0$ and $|2\lambda e_N-y| \le |y|$ for $y\in \mathbb R^N_+
\setminus \Sigma_{2\lambda}$, we deduce $u\equiv 0$ in $\mathbb R^N_+ \setminus \Sigma_{2\lambda}$.

Using this information and evaluating the equation at points $x\in \mathbb R^N_+\setminus \Sigma_{2\lambda}$,
we obtain
\begin{equation}\label{eq3}
\int_{\Sigma_{2\lambda}} \frac{u(y)}{|x-y|^{N+2s}} dy =-f(0).
\end{equation}
Now observe that the integral above is a smooth function of $x$ if, say, $x_N\ge 2\lambda+1$, since
the integrand is uniformly bounded and the integral is uniformly convergent at infinity when $x$ belongs to a
compact set. Therefore, we are allowed to differentiate \eqref{eq3} with respect to $x_N$ to get:
$$
\int_{\Sigma_{2\lambda}} \frac{u(y)(x_N-y_N)}{|x-y|^{N+2s+2}} dy =0.
$$
However, $x_N-y_N\ge x_N-2\lambda\ge 1$ for $y\in \Sigma_{2\lambda}$ and the chosen values of $x$, so that the
integrand is nonnegative and this gives $u\equiv 0$ in $\Sigma_{2\lambda}$, therefore in
$\mathbb R^N$. It is clear that this can only happen when $f(0)=0$, and the proof is concluded.
\end{proof}

\bigskip

\section{Proof of the main results}
\setcounter{section}{5}
\setcounter{equation}{0}

In this final section we will prove our main contributions, Theorems \ref{teorema-1} and
\ref{teorema-2}. The proof of Corollary \ref{coro-lineal} will not be given, since it is an
immediate consequence of Theorem \ref{teorema-2}.

\bigskip

\begin{proof}[Proof of Theorem \ref{teorema-1}]
We have already said that the proof is an application of the method of moving planes, as used in
\cite{QX} and \cite{FW}, but with some significant changes. In particular, we remark that we work
with some truncations of the original functions, so we are led to the use of inequalities in the viscosity
sense and Lemma \ref{lema-varianteFW}. We also need at some point the nonexistence result given
by Theorem \ref{lema-noexistencia}.

We follow the notation introduced in  Section 2.

\medskip

\noindent {\bf Step 1}. $w_\lambda \ge 0$ in $\Sigma_\lambda$ if $\lambda>0$ is small enough.

\smallskip

To prove this, assume for a contradiction that $D_\lambda$ is not empty if $\lambda$ is small.  Since $u$
is bounded and $f$ is $C^1$, there exists a constant $L$ such that $f(u^\lambda)-f(u)\ge -L |u^\lambda-u|$.
Therefore, using Lemma \ref{lema-signo} we have
$$
(-\Delta)^s v_\lambda \ge f(u^\lambda)-f(u) \ge -L |u^\lambda-u|= L v_\lambda \quad \hbox{in }
D_\lambda.
$$
By Theorem 2.4 in \cite{QX} we obtain $v_\lambda\ge 0$ in $D_\lambda$ when $\lambda$ is small enough,
which is a contradiction. Therefore, $D_\lambda=\emptyset$ for small $\lambda$ and this shows the claim.

\medskip

\noindent {\bf Step 2.} Setting
$$
\lambda^*=\sup\{ \lambda>0: \ w_\mu\ge 0 \hbox{ in } \Sigma_\mu \hbox{ for every } \mu \in (0,\lambda)\},
$$
we have $\lambda^*=+\infty$.

\smallskip

Assume again for a contradiction that
$\lambda^*<+\infty$. Then there exists a sequence $\{\lambda_j\}$ of values such that $\lambda_j>\lambda^*$ for every
$j$ and $\lambda_j\to \lambda^*$ as $j\to +\infty$, with $w_{\lambda_j}$ negative somewhere in $\Sigma_{\lambda_j}$.
Consider the sets
$$
D_j= \{x\in \Sigma_{\lambda_j}: \ w_{\lambda_j} (x)<0\}
$$
and
$$
W_j=\{x\in D_j:\ f(u(x))> f(u^{\lambda_j}(x))\}.
$$
By the choice of $\lambda_j$, the sets $D_j$ are nonempty for every $j$. We claim that the
same is true for $W_j$. Indeed, if we had $W_j=\emptyset$, then $f(u^{\lambda_j})\ge f(u)$
in $D_j$. Hence
$$
(-\Delta)^s v_{\lambda_j} \ge 0 \quad \hbox{in } D_j.
$$
By Lemma \ref{lema-pmax} we obtain $v_{\lambda_j}\ge 0$ in $D_j$, which is not possible.
Therefore $W_j\neq \emptyset$. Thus it is possible to choose points $x_j\in W_j$ such that
\begin{equation}\label{eq-eleccion}
v_{\lambda_j}(x_j) \le -\frac{1}{2} \| v_{\lambda_j} \|_{L^\infty(W_j)},
\end{equation}
and we can define the functions
$$
\widetilde u_j(x)= u(x'+x_j',x_N), \quad x\in \mathbb R^N_+.
$$
It is easily seen that  $\widetilde u_j$ is a solution of \eqref{problema}, verifying $\| \widetilde u_j \|_{L^\infty(\mathbb R^N_+)}=
\| u \|_{L^\infty(\mathbb R^N_+)}$. It is then standard, with the use of regularity theory, Ascoli-Arzel\'a's
theorem and a diagonal argument, that for some subsequence
$$
\widetilde u_j \to \bar u
$$
locally uniformly in $\mathbb R^N$, where $\bar u$ is a nonnegative solution of
\eqref{problema}. We may also assume that $x_{j,N}\to x_0\in [0,\lambda^*]$. Now three cases are
possible:

\begin{itemize}

\item[(a)] $\bar u\not \equiv 0$, $x_0\in (0,\lambda^*)$;

\smallskip
\item[(b)] $\bar u\not \equiv 0$, $x_0=0$ or $x_0=\lambda^*$;

\smallskip
\item[(c)] $\bar u \equiv 0$.

\end{itemize}

\medskip
\noindent Before dealing with each of this cases, let us introduce some notation related to the
functions $\widetilde u_j$. Let:
$$
\begin{array}{l}
\widetilde w_{\lambda_j}(x)= \widetilde u_j (x^{\lambda_j})- \widetilde u_j(x), \ \ x\in \Sigma_{\lambda_j},\\[0.5pc]
\widetilde{D}_j=\{x\in \Sigma_{\lambda_j}: \ \widetilde w_j(x)<0\},\\[0.5pc]
\widetilde{v}_{\lambda_j}(x) = \widetilde w_{\lambda_j}(x) \chi_{\widetilde D_j}(x), \ \ x\in \Sigma_{\lambda_j}, \\[0.5pc]
\widetilde{W}_j=\{x\in \widetilde{D}_j:\ f(\widetilde u_j(x))> f(\widetilde u_j(x^\lambda_j))\}
\end{array}
$$
(observe that $\widetilde{D}_j$ and $\widetilde{W}_j$ are nothing more than translations
of $D_j$ and $W_j$, respectively). Denote also $z_j=(0,x_{j,N})$, $z_0=(0,x_0)$.
By our choice of $x_j$ in \eqref{eq-eleccion} above, since it follows that $\widetilde{w}_{\lambda_j} (z_j)<0$ and $f(\widetilde u_j(z_j))> f(\widetilde u_j(z_j^{\lambda_j}))$, we deduce that $z_j \in \widetilde{W}_j$. Moreover we also get
\begin{equation}\label{eq-eleccion-2}
\widetilde v_{\lambda_j}(z_j) \le -\frac{1}{2} \| \widetilde v_{\lambda_j}\|_{L^\infty(\widetilde{W}_j)}.
\end{equation}
Now consider in turn each one of the cases (a), (b) and (c).

\medskip

In case (a), we see from $\widetilde u_j(x^{\lambda^*})\ge \widetilde u_j(x)$ in $\Sigma_{\lambda^*}$ that
$\bar u(x^{\lambda^*}) \ge \bar u(x)$ in $\Sigma_{\lambda^*}$. Moreover, since $\widetilde u(z_j^{\lambda_j}) <
\widetilde u(z_j)$,  we also have $\bar u(z_0^{\lambda^*}) = \bar u(z_0)$.
Let us see that this implies $\bar u(x^{\lambda^*})\equiv \bar u(x)$ in $\mathbb R^N$. Indeed, arguing as in the
proof of Lemma \ref{lema-signo} and denoting $\bar w_{\lambda^*}=\bar u ^{\lambda^*}-\bar u$, we obtain
\begin{align*}
0 & =f(\bar u^{\lambda^*}(z_0))-f(\bar u(z_0))= (-\Delta)^s \bar w_{\lambda^*}(z_0)\\
& =-  \int_{\Sigma_{\lambda^*} \cup \mathbb R^N_-} \bar w_{\lambda^*}(y) \left( \frac{1}{|z_0-y|^{N+2s}} - \frac{1}{|z_0-y^\lambda|^{N+2s}} \right)dy,
\end{align*}
which implies $\bar w_{\lambda^*}\equiv 0$ in $\mathbb R^N$, since $\bar w_{\lambda^*}\ge 0$ in $\Sigma_{\lambda^*} \cup \mathbb R^N_-$ and
$|z_0-y|\le |z_0-y^\lambda|$ for every $y\in \Sigma_{\lambda^*} \cup \mathbb R^N_-$.

This means that $\bar u$ is symmetric with respect to $T_{\lambda^*}$, so that Theorem \ref{lema-noexistencia}
implies $\bar u\equiv 0$,  a contradiction.

\medskip

As for case (b), assume $x_0=0$. We deduce from Lemma \ref{lema-varianteFW} and the choice of the points
$z_j$:
\begin{align*}
\frac{1}{2} \| \widetilde v_{\lambda_j}\|_{L^\infty(\widetilde{W}_j)} \le - \widetilde v_{\lambda_j}(z_j) &
\le L  \| \widetilde v_{\lambda_j}\|_{L^\infty(\widetilde{W}_j)} \int_{\widetilde{W}_j} G_\infty^+(z_j,y) dy\\[1pc]
& \le L  \| \widetilde v_{\lambda_j}\|_{L^\infty(\widetilde{W}_j)} \int_{\Sigma_{\lambda^*+1}} G_\infty^+(z_j,y) dy,
\end{align*}
where $L$ denotes a bound for the derivative of $f$ in $[0,\|u\|_{L^\infty(\mathbb R^N_+)}]$.
By Lemma \ref{lema-FW}, part (c), the last integral converges to zero since $z_j\to 0\in \partial \mathbb R^N_+$.
We deduce that $\widetilde v_{\lambda_j}=0$ in $\widetilde{W}_j$ when $j$ is large enough, a contradiction.

When $x_0=\lambda^*$, a similar contradiction is reached. The only difference is that one now
works with the Green's function in the half-space $\{x\in \mathbb R^N:\ x_N<\lambda_j\}$ (see Remark
\ref{rem-semiespacio}).

\bigskip

Finally, we consider case (c). This case can only arise when $f(0)=0$ and the proof is
different depending on the sign of $f'(0)$. We begin by assuming that $f'(0)>0$.

In what follows, we denote by $\lambda_1(\Omega)$ the principal eigenvalue of $(-\Delta)^s$ in $\Omega$ under
Dirichlet boundary conditions (cf. Proposition 9 in \cite{SV2}). If we take a ball $B_R$ with arbitrary
center and radius $R$ then it can be seen by means of a simple scaling that
$$
\lambda_1(B_R)=\frac{\lambda_1(B_1)}{R^{2s}}\to 0 \quad \hbox{as } R\to +\infty.
$$
Thus it is possible to select a ball $B\subset \subset \mathbb R^N_+$ with the property that
\begin{equation}\label{eq-autovalor}
\lambda_1(B)<\frac{1}{2} f'(0).
\end{equation}
Since $\widetilde u_j\to 0$ uniformly in $B$, we deduce
\begin{equation}\label{eq-ultima}
(-\Delta)^s \widetilde u_j = \frac{f(\widetilde u_j)}{\widetilde u_j}\; \widetilde u_j\ge \frac{1}{2}f'(0) \;
\widetilde u_j \quad \hbox{in }B
\end{equation}
if $j$ is large enough. By Theorem 1.1 in \cite{QSX},  \eqref{eq-ultima} implies the opposite inequality in
\eqref{eq-autovalor}, which is a contradiction.

Hence to conclude the proof only the case $f'(0)\le 0$ needs to be dealt with. Using the
mean value theorem, we may write, for $y\in \widetilde W_j$:
$$
f(\widetilde u_j^{\lambda_j}(y))- f(\widetilde u_j(y)) = f'(\xi_j(y)) \widetilde v_j(y),
$$
where $\xi_j(y)$ is an intermediate value between $\widetilde u_j(y)$ and $\widetilde u_j^{\lambda_j}(y)$.
Observe that $\xi_j\to 0$ uniformly on compact sets of $\mathbb R^N_+$ while
$f'(\xi_j)\ge 0$. By Lemma \ref{lema-varianteFW} and \eqref{eq-eleccion-2}, we see that
\begin{equation}\label{eq5}
\frac{1}{2} \| \widetilde v_{\lambda_j}\|_{L^\infty(\widetilde{W}_j)} \le  \| \widetilde v_{\lambda_j}\|_{L^\infty(\widetilde{W}_j)}
\int_{\Sigma_{\lambda^*+1}} G_\infty^+ (z_j,y) f'(\xi_j(y)) dy.
\end{equation}
Now choose $R>0$ and split the integral in \eqref{eq5} in $B_R$ and $B_R^c$. If
$L$ stands again for a bound for the derivative of $f$ in $[0,\|u\|_{L^\infty(\mathbb R^N_+)}]$, we have
\begin{align*}
\frac{1}{2} \| \widetilde v_{\lambda_j}\|_{L^\infty(\widetilde{W}_j)} & \le \| \widetilde v_{\lambda_j}\|_{L^\infty(\widetilde{W}_j)}
\left(\int_{\Sigma_{\lambda^*+1} \cap B_R} G_\infty^+ (z_j,y)  f'(\xi_j(y)) dy \right.\\[1pc]
& \left. + L \int_{\Sigma_{\lambda^*+1} \cap B_R^c} G_\infty^+ (z_j,y) dy\right).
\end{align*}
Observe that the integral in $\Sigma_{\lambda^*+1}\cap B_R^c$ can be made as small as
desired by taking $R$ large enough, thanks to the fact that $\{z_j\}$ is a bounded sequence
and Lemma \ref{lema-FW}, part (a). Therefore the last term in the
above inequality can be made less than $\frac{1}{4}$, say, if $R$ is chosen large.
After we have fixed such a value of $R$, we observe that $\xi_j \to 0$ uniformly in
$B_R$, so that $f'(\xi_j)\le o(1)$, since we are assuming $f'(0)\le 0$.
Therefore, using Lemma \ref{lema-FW}, part (b), we arrive at
$$
\frac{1}{4} \| \widetilde v_{\lambda_j}\|_{L^\infty(\widetilde{W}_j)} \le o(1) \| \widetilde v_{\lambda_j}\|_{L^\infty(\widetilde{W}_j)}
$$
which, as above, is a contradiction.

\medskip

\noindent {\bf Step 3}. Proof of \eqref{eq-derivadapos}.

\smallskip

As a consequence of steps 1 and 2, we have shown that $u^\lambda\ge u$ in $\Sigma_\lambda$ for every
$\lambda>0$, that is, $u$ is nondecreasing as a function of the variable $x_N$. Since $u\in C^1(\mathbb R^N_+)$,
this implies
\begin{equation}\label{eq-der}
\frac{\partial u}{\partial x_N} \ge 0 \quad \hbox{in } \mathbb R^N_+.
\end{equation}
To conclude the proof of our theorem, we only have to show that the inequality is strict in
\eqref{eq-der}. This is a consequence of the strong maximum principle for the derivative
with respect to $x_N$. However, this function does not directly verify an equation in $\mathbb R^N_+$,
since $u$ is not expected to be $C^1$ on $\partial \mathbb R^N_+$. We overcome this difficulty by
localizing the problem and working with incremental quotients.

Assume there exists $x_0\in \mathbb R^N_+$ such that
\begin{equation}\label{eq-derivadacero}
\frac{\partial u}{\partial x_N} (x_0)=0.
\end{equation}
Choose $\delta>0$ such that $B_{2\delta} (x_0)\subset \subset \mathbb R^N_+$, and let $\phi \in C_0^\infty
(B_{2\delta} (x_0))$ be a cut-off function with the usual properties: $0\le \phi \le 1$ and $\phi=1$ in
$B_{\delta} (x_0)$. Choose a small $\tau>0$ and define for $x\in \mathbb R^N$:
$$
z_\tau(x)= \frac{u(x+\tau e_N)-u(x)}{\tau} \phi(x).
$$
Since $u$ is nondecreasing we have $z_\tau \ge 0$, and we obtain
$$
(-\Delta)^s z_\tau \ge \frac{f(u(x+\tau e_N))-f(u(x))}{\tau} \quad \hbox{in } B_\delta(x_0),
$$
with $z_\tau \ge 0$ in $\mathbb R^N$. Letting $\tau \to 0$, it is clear that
$$
z_t\to z:= \frac{\partial u}{\partial x_N} \phi \quad \hbox{uniformly in }\mathbb R^N,
$$
and using Lemma 4.5 in \cite{CS} we see that
$$
(-\Delta)^s z \ge f'(u)z \quad \hbox{in } B_\delta(x_0),
$$
in the viscosity sense. The strong maximum principle and \eqref{eq-derivadacero} imply $z=0$ in
$B_\delta (x_0)$, so that $\frac{\partial u}{\partial x_N}=0$ in $B_\delta(x_0)$. A standard connectedness argument
then implies that
$$
\frac{\partial u}{\partial x_N}=0 \quad \hbox{in } \mathbb R^N_+,
$$
which is impossible. This concludes the proof of \eqref{eq-derivadapos}. Observe by the way
that $u>0$ in $\mathbb R^N_+$ is a direct consequence of \eqref{eq-derivadapos}.
\end{proof}

\bigskip

\begin{proof}[Proof of Theorem \ref{teorema-2}]
Assume $u$ is a nonnegative, bounded, nontrivial solution of \eqref{problema}. By Theorem
\ref{teorema-1}, we have $u>0$ in $\mathbb R^N_+$. We first claim that for every $\delta>0$, there exists $c(\delta)>0$
such that
$$
u(x) \ge c(\delta) \quad \hbox{if } x_N\ge \delta.
$$
Suppose for a contradiction that this is not true. Then there exists $\delta>0$ and a sequence
$\{x_n'\}\subset \mathbb R^{N-1}$ such that $u(x_n',\delta)\to 0$. Define
$$
u_n(x) = u(x'+x_n',x_N), \quad x\in \mathbb R^N_+.
$$
Since $\{u_n\}$ is uniformly bounded, we obtain after passing to a subsequence that $u_n\to \bar u$
locally uniformly in $\mathbb R^N$,
where $\bar u$ is a nonnegative, bounded solution of \eqref{problema} which verifies $\bar u(0,\delta)=0$.
Again by Theorem \ref{teorema-1} we see that $\bar u \equiv 0$.

This is impossible if $f$ verifies (a) in the statement. When $f$ verifies (b), a similar argument as in
case (c) in step 2 of the proof of Theorem \ref{teorema-1} also leads to a contradiction. This
contradiction proves the claim.

Now fix any $\delta>0$ and let
$$
\theta =\inf_{c(\delta)\le t \le M} f'(t)>0,
$$
where $M= \| u\|_{L^\infty(\mathbb R^N_+)}$. If we choose any function $\phi \in C_0^\infty (\mathbb R^N_+\setminus
\Sigma_\delta)$ such that $0\le \phi\le 1$ and $\phi=1$ in $\mathbb R^N\setminus \Sigma_{2\delta}$, we see
as in step 3 in the proof of Theorem \ref{teorema-1} that the function
$$
z= \frac{\partial u}{\partial x_N} \phi
$$
verifies
$$
(-\Delta)^s z \ge \theta z \quad \hbox{in }\mathbb R^N\setminus \Sigma_{2\delta}.
$$
Arguing again with the principal eigenvalue $\lambda_1(B)$ in a sufficiently large ball $B$ contained in
$\mathbb R^N\setminus \Sigma_{2\delta}$ we reach a contradiction. This shows that no bounded, nonnegative,
nontrivial solution to \eqref{problema} may exist under our hypotheses.
\end{proof}

\bigskip

\noindent {\bf Acknowledgements.} B. B. was partially supported by MEC Juan de la Cierva postdoctoral fellowship
number  FJCI-2014-20504 (Spain).
L. D. P. was partially supported by PICT2012 0153 from ANPCyT (Argentina).
B. B., J. G-M. and A. Q. were partially supported by Ministerio de Ciencia e
Innovaci\'on under grant MTM2014-52822-P (Spain). A. Q. was also partially supported by Fondecyt Grant
No. 1151180 Programa Basal, CMM. U. de Chile and Millennium Nucleus
Center for Analysis of PDE NC130017. B. B.  and J. G-M. would like to thank the Mathematics
Department of Universidad T\'ecnica Federico Santa Mar\'ia where part of this work has been done
for its kind hospitality.