$\ $ **Efficient Estimation of Partially Linear Models for Spatial Data over Complex Domains** Li Wang$^{a}$, Guannan Wang$^{b}$, Min-Jun Lai$^{c}$ and Lei Gao$^{a}$ [^1] *$^{a}$Iowa State University, $^{b}$College of William & Mary and $^{c}$University of Georgia* > *Abstract:* In this paper, we study the estimation of partially linear models for spatial data distributed over complex domains. We use bivariate splines over triangulations to represent the nonparametric component on an irregular two-dimensional domain. The proposed method is formulated as a constrained minimization problem which does not require constructing finite elements or locally supported basis functions. Thus, it allows an easier implementation of piecewise polynomial representations of various degrees and various smoothness over an arbitrary triangulation. Moreover, the constrained minimization problem is converted into an unconstrained minimization via a QR decomposition of the smoothness constraints, which leads to a penalized least squares method to estimate the model. The estimators of the parameters are proved to be asymptotically normal under some regularity conditions. The estimator of the bivariate function is consistent, and its rate of convergence is also established. The proposed method enables us to construct confidence intervals and permits inference for the parameters. The performance of the estimators is evaluated by two simulation examples and by a real data analysis. > > *Key words and phrases:* Bivariate splines, Penalty, Semiparametric regression, Spatial data, Triangulation. \[SEC:introduction\] **1. Introduction** 0.1in In many geospatial studies, spatially distributed covariate information is available. For example, geographic information systems may contain measurements obtained from satellite images at some locations. These spatially explicit data can be useful in the construction and estimation of regression models, but sometimes they are distributed over irregular two-dimensional (2-D) domains that may have complex boundaries or holes inside. It is well known that many conventional smoothing tools suffer from the problem of "leakage" across the complex domains, which refers to the poor estimation over difficult regions by smoothing inappropriately across boundary features, such as peninsulas; see excellent discussions in Ramsay (2002) and Wood, Bravington and Hedley (2008). In this paper, we propose to use bivariate splines (smooth piecewise polynomial functions over a triangulation of the domain of interest) to model spatially explicit datasets which enable us to overcome the "leakage" problem and find the estimation more accurately. We focus here on the partially linear model (PLM), popularized by Hardle, Liang and Gao (2000), for data randomly distributed over 2-D domains. To be more specific, let $\mathbf{X}_{i}=(X_{i1},X_{i2})^{\T}$ be the location of $i$-th point, $i=1,\ldots, n$, which ranges over a bounded domain $\Omega \subseteq \mathbb{R}^2$ of arbitrary shape, for example, a domain with polygon boundary. Let $Y_{i}$ be the response variable and $\mathbf{Z}_{i}=(Z_{i1},\ldots,Z_{ip})^{\T}$ be the predictors at location $\mathbf{X}_{i}$. Suppose that $\left\{(\mathbf{Z}_{i},\mathbf{X}_{i},Y_{i})\right\} _{i=1}^{n}$ satisfies the following model $$Y_{i}=\mathbf{Z}_i^{\T} \boldsymbol{\beta}+ g\left( \mathbf{X}_{i}\right) + \epsilon _{i}, \quad i=1,\ldots ,n, \label{model}$$ where $\boldsymbol{\beta}=(\beta_{1},\ldots,\beta_{p})^{\T}$ are unknown parameters, $g(\cdot)$ is some unknown but smooth bivariate function, and $\epsilon_{i}$'s are i.i.d random noises with $E\left(\epsilon _{i}\right)=0$ and $\mathrm{Var}\left( \epsilon _{i}\right) =\sigma^2$. Each $\epsilon_{i}$ is independent of $\mathbf{X}_{i}$ and $\mathbf{Z}_{i}$. In many situations, our main interest is in estimating and making inference for the regression parameters $\boldsymbol{\beta}$, which provides measures of the effect of the covariate $\mathbf{Z}$ after adjusting for the location effect of $\mathbf{X}$. If $g(\cdot)$ is a univariate function, model (\[model\]) becomes a typical PLM. In the past three decades, flexible and parsimonious PLMs have been extensively studied and widely used in many statistical applications, from biostatistics to econometrics, from engineering to social science; see Chen, Liang and Wang (2011), Huang, Zhang and Zhou (2007), Liang and Li (2009), Liu, Wang and Liang (2011), Ma, Song and Wang (2013), Ma and Yang (2011), Wang, et al (2011), Wang, et al (2014), Zhang, Cheng and Liu (2011) for some recent works on PLMs. When $g(\cdot)$ is a bivariate function, there are two popular estimation tools: bivariate P-splines Marx and Eilers (2005) and thin plate splines Wood (2003). Later, Xiao, Li and Ruppert (2013) proposed an efficient sandwich smoother, which has a tensor product structure that simplifies an asymptotic analysis and can be fast computed. Their method has been applied to quantifying the lifetime circadian rhythm of physical activity Xiao, et al (2015). The application to spatial data analysis over complex domains, however, has been hampered due to the scarcity of bivariate smoothing tools that are not only computationally effficient but also theoretically reliable to solve the problem of "leakage" across the domain. Traditional smoothing methods in practical data analysis, such as kernel smoothing, wavelet-based smoothing, tensor product splines and thin plate splines, usually perform poorly for those data, since they do not take into account the shape of the domain and also smooth across concave boundary regions. There are several challenges when going from rectangular domains to irregular domains with complex boundaries or holes. Some efforts have recently been devoted to studying the smoothing over irregular domains, and significant progress has been made. Most of them are based on the roughness penalization approach Green and Silverman (1994). To deal with irregular domains, Wang and Ranalli (2007) applied low-rank thin-plate splines defined as functions of the geodesic distance instead of the Euclidean distance. Eilers (2006) utilized the Schwarz-Christoffel transform to convert the complex domains to regular domains. To solve the "leakage" problem, in a pioneering paper, Ramsay (2002) suggested a penalized least squares approach with a Laplacian penalty and transformed the problem to that of solving a system of partial differential equations (PDEs). Wood, Bravington and Hedley (2008) provided an elegant solution and developed the soap film smoothing estimator for smoothing over difficult regions that can be represented by a low-rank basis and one or two quadratic penalties. Recently, Sangalli, Ramsay and Ramsay (2013) extended the method in Ramsay (2002) to the PLMs, which allows for spatially distributed covariate information to be in the models. The data smoothing problem in Sangalli, Ramsay and Ramsay (2013) is solved using finite element method (FEM), a method mainly developed and used to solve PDEs. Although their method is practically useful, the theoretical properties of the estimation are not investigated. In this paper, we tackle the estimation problem differently from Sangalli, Ramsay and Ramsay (2013). Instead of using FEM, we approximate the nonparametric function $g(\cdot)$ by using the bivariate splines over triangulations in Lai and Schumaker (2007). An important feature of this approach is that it uses splines for applications without constructing locally supported splines or finite elements and without computing the dimension. This method has been shown to be more efficient and flexible than the conventional FEM in data fitting problems and solving PDEs; see Awanou, Lai and Wenston (2005), Ettinger, Guillas and Lai (2015), Guillas and Lai (2010), Lai and Wang (2013) and Liu, Guillas and Lai (2015). For example, the users can choose spline functions of flexible degrees and various smoothness across any given domain. Another advantage is that the linear systems arising in this approach are more easy to assemble than those from the finite elements or locally supported spline basis functions. The linear systems are sparser than that from any macro-FEM method. In addition, due to the great scalability, the assembling can be computed in parallel. To the best of the authors' knowledge, statistical aspects of smoothing for PLMs by using bivariate splines have not been discussed in the literature so far. This paper presents the first attempt at investigating the asymptotic properties of the PLMs for data distributed on a non-rectangular complex region. We study the asymptotic properties of the least squares estimators of $\boldsymbol{\beta}$ and $g(\cdot)$ by using bivariate splines over triangulations with a penalty term. We show that our estimator of $\boldsymbol{\beta}$ is root-$n$ consistent and asymptotically normal, although the convergence rate of the estimator of the nonparametric component $g(\cdot)$ is slower than root-$n$. A standard error formula for the estimated coefficients is provided and tested to be accurate enough for practical purposes. Hence, the proposed method enables us to construct confidence intervals for the regression parameters. We also obtain the convergence rate for the functional estimator of $g(\cdot)$. We show, by using numerical studies, that our method is competitive with existing methods such as the soap lm smoother and the thin-plate regression spline. The rest of the paper is organized as follows. In Section 2, we give a brief review of the triangulations and propose our estimation method based on penalized bivariate splines. We also discuss the details on how to choose the penalty parameters. Section 3 is devoted to the asymptotic analysis of the proposed estimators. Section 4 provides a detailed simulation to compare several methods in two different scenarios and explores the estimation and prediction accuracy. Section 5 studies a real dataset on house values over California. Some concluding remarks are given in Section 6. Technical details are provided in Appendix A. .12in **2. Triangulations and Penalized Spline Estimators** \[sec:spline\] Our estimation method is based on penalized bivariate splines over triangulations. The idea is to approximate the function $g(\cdot)$ by bivariate splines that are piecewise polynomial functions over a 2D triangulated domain. We use this approximation to construct least squares estimators of the linear and nonlinear components of the model with a penalization term. In the following of this section, we describe the background of triangulations, B-form bivariate splines and introduce the penalized spline estimators. .10in **2.1. Triangulations** .10in Triangulation is an effective strategy to handle data distribution on irregular regions with complex boundaries and/or interior holes. Recently, it has attracted substantial recent attention in many applied areas, such as geospatial studies, numerical solutions of PDEs, image enhancements, and computer aided geometric design. See, for example, the recent comprehensive book by Lai and Schumaker (2007) and the article by Lai (2008). We use $\tau$ to denote a triangle which is a convex hull of three points not located in one line. A collection $\triangle=\{\tau_1,...,\tau_{N}\}$ of $N$ triangles is called a triangulation of $\Omega=\cup_{i=1}^{N} \tau_i$ provided that if a pair of triangles in $\triangle$ intersect, then their intersection is either a common vertex or a common edge. In general, any kind of polygon shapes can be used for the partition of $\Omega$. In this paper we consider triangulations of $\Omega$ because any polygonal domain of arbitrary shape can be partitioned into finitely many triangles. In the following, we assume that all ${\bf X}_i$s are inside triangles of $\triangle$. That is, they are not on edges or vertices of triangles in $\triangle$. Otherwise, we can simply count them twice or multiple times if any observation is located on an edge or at a vertex of $\triangle$. Given a triangle $\tau\in \triangle$, we let $|\tau|$ be its longest edge length, and denote the size of $\triangle$ by $|\triangle |=\max \{|\tau|,\tau\in \triangle \}$, i.e., the length of the longest edge of $\triangle$. Furthermore, let $\rho_{\tau}$ be the radius of the largest circle inscribed in $\tau$. We measure the quality of a triangulation $\triangle$ by $\delta_{\triangle}=\max_{\tau\in\triangle}{|\tau|}/{\rho_{\tau}}<\infty$, which is equivalent to the smallest angle of $\triangle$. The study in Lai and Schumaker (2007) shows that the approximation of spline spaces over $\triangle$ is dependent on $\delta_{\triangle}$, i.e., the larger the $\delta_{\triangle}$ is, the worse the spline approximation is. In the rest of the paper, we restrict our attention to the triangulations satisfying $\delta_{\triangle}<\delta$ for a positive constant $\delta$. .12in **2.2. B-form bivariate splines** .10in In this section we give a brief introduction to the bivariate splines. More in-depth description can be found in Lai and Schumaker (2007), Lai (2008), as well as Zhou and Pan (2014) and the details of the implementation is provided in Awanou, Lai and Wenston (2005). Let $\tau = \langle \mathbf{v}_{1}, \mathbf{v}_{2}, \mathbf{v}_{3} \rangle$ be a non-degenerate (i.e. with non-zero area) triangle with vertices $\mathbf{v}_{1}$, $\mathbf{v}_{2}$, and $\mathbf{v}_{3}$. Then for any point $\mathbf{v}\in\mathbb{R}$, there is a unique representation in the form $$\mathbf{v} = b_1\mathbf{v}_1 + b_2\mathbf{v}_2 + b_3\mathbf{v}_3$$ with $b_1 + b_2 + b_3 = 1$, where $b_1$, $b_2$ and $b_3$ are called the barycentric coordinates of the point $\mathbf{v}$ relative to the triangle $\tau$. The Bernstein polynomials of degree $d$ relative to triangle $\tau$ is defined as $$B_{ijk}^{\tau,d}(\mathbf{v})=\frac{d}{i!j!k!}b_1^ib_2^jb_3^k.$$ Then for any $\tau\in \triangle$, we can write the polynomial piece of spline $s$ restricted on $\tau\in \triangle$ as $$s|_{\tau}=\sum_{i+j+k=d} \gamma_{ijk}^{\tau}B_{ijk}^{\tau,d},$$ where the coefficients $\boldsymbol{\gamma}_{\tau}=\{\gamma_{ijk}^{\tau}, i+j+k=d\}$ are called B-coefficients of $s$. For a nonnegative integer $r$, let $\mathbb{C}^r(\Omega)$ be the collection of all $r$-th continuously differentiable functions over $\Omega$. Given a triangulation $\triangle$, let $\mathbb{S}_{d}^{r}(\triangle)=\{s\in \mathbb{C}^{r}(\Omega):s|_{\tau}\in \mathbb{P}_{d}(\tau), \tau \in \triangle \}$ be a spline space of degree $d$ and smoothness $r$ over triangulation $\triangle $, where $\mathbb{P}_{d}$ is the space of all polynomials of degree less than or equal to $d$. For notation simplicity, let $\mathbb{S}=\mathbb{S}_{3r+2}^{r}(\triangle)$ for a fixed smoothness $r\ge 1$, and we know that such a spline space has the optimal approximation order (rate of convergence) for noise-free datasets; see Lai and Schumaker (1998) and Lai and Schumaker (2007). For notation simplicity, let $\{B_{\xi}\}_{\xi \in \mathcal{K}}$ be the set of degree-$d$ bivariate Bernstein basis polynomials for $\mathbb{S}$, where $\mathcal{K}$ stands for an index set of $K$ Bernstein basis polynomials. Then for any function $s\in \mathbb{S}$, we can represent it by using the following basis expansion: $$s(\mathbf{x})=\sum_{\xi \in \mathcal{K}} B_{\xi}(\mathbf{x})\gamma_{\xi} =\mathbf{B}(\mathbf{x})^{\T}\boldsymbol{\gamma}, \label{EQ:basis-expansion}$$ where $\boldsymbol{\gamma}^{\T} =(\gamma_{\xi},\xi \in \mathcal{K})$ is the spline coefficient vector. To meet the smoothness requirement of the splines, we need to impose some linear constraints on the spline coefficients $\boldsymbol{\gamma}$ in (\[EQ:basis-expansion\]). We require that $\boldsymbol{\gamma}$ satisfies $\mathbf{H}\boldsymbol{\gamma}=0$ with $\mathbf{H}$ being the matrix for all smoothness conditions across shared edges of triangles, which depends on $r$ and the structure of the triangulation. 0.10in **2.3. Penalized Spline Estimators** 0.10in To define the penalized spline method, for any direction $x_{j}$, $j=1,2$, let $D_{x_{j}}^{q}f(\mathbf{x})$ denote the $q$-th order derivative in the direction $x_{j}$ at the point $\mathbf{x}=(x_1,x_2)$. Let $$\mathcal{E}_{\upsilon }(f)= \sum_{\tau\in \triangle}\int_{\tau} \sum_{i+j=\upsilon } \binom{\upsilon}{i} (D_{x_{1}}^{i}D_{x_{2}}^{j}f)^{2}dx_{1}dx_{2} \label{penalty}$$ be the energy functional for a fixed integer $\upsilon \ge 1$. Although all partial derivatives up to the chosen order $\upsilon$ can be included in the penalty of (\[penalty\]), for simplicity, in the remaining part of the paper, we use $\upsilon =2$, and one can study the similar problem for general $\upsilon \geq 2$. When $\upsilon=2$, $$\mathcal{E}_{2}(f)=\int_{\Omega} \left((D_{x_{1}}^{2}f)^2+2(D_{x_{1}}D_{x_{2}}f)^2 +(D_{x_{2}}^{2}f)^2\right) dx_{1}dx_{2}, \label{penalty-2}$$ which is similar to the thin-plate spline penalty (Green and Silverman, 1994) except the latter is integrated over the entire plane $\mathbb{R}^{2}$. Sangalli, Ramsay and Ramsay (2013) used a different roughness penalty from (\[penalty-2\]), specifically, they use the integral of the square of the Laplacian of $f$, that is, $\lambda\int_{\Omega}(D_{x_{1}}^{2}f+D_{x_{2}}^{2}f)^{2}dx_{1}dx_{2}$. Both forms of penalties are invariant with respect to Euclidean transformations of spatial co-ordinates, thus, the bivariate smoothing does not depend on the choice of the coordinate system. Given $\lambda >0$ and $\{(\mathbf{Z}_i,\mathbf{X}_i,Y_i)\}_{i=1}^{n}$, we consider the following minimization problem: $$\begin{aligned} \min_{s\in \mathbb{S}}\sum_{i=1}^{n}\left\{Y_{i}-\mathbf{Z}_{i}^{\T} \boldsymbol{\beta} -s\left( \mathbf{X}_{i}\right)\right\}^{2}+\lambda \mathcal{E}_{\upsilon }(s). \label{DEF:minimization}\end{aligned}$$ Let $\mathbf{Y} = (Y_1,\ldots , Y_n)^{\T}$ be the vector of $n$ observations of the response variable. Denote by $\mathbf{X} _{n\times 2}= \{(X_{i1}, X_{i2})\}_{i=1}^{n}$ the location design matrix and $\mathbf{Z}_{n\times p}= \{(Z_{i1}, \ldots,Z_{ip})\}_{i=1}^{n}$ the collection of all covariates. Denote by $\mathbf{B}$ the $n\times K$ evaluation matrix of Bernstein basis polynomials whose $i$-th row is given by $\mathbf{B}_{i}^{\T}=\{B_{\xi}(\mathbf{X}_{i}), \xi\in \mathcal{K}\}$. Then the minimization problem in (\[DEF:minimization\]) reduces to $$\min_{\boldsymbol{\beta},\boldsymbol{\gamma}} L(\boldsymbol{\beta},\boldsymbol{\gamma})=\min_{\boldsymbol{\beta},\boldsymbol{\gamma}} \left\{\|\mathbf{Y}-\mathbf{Z}\boldsymbol{\beta}-\mathbf{B}\boldsymbol{\gamma}\|^{2}+\lambda\boldsymbol{\gamma}^{\T}\mathbf{P}\boldsymbol{\gamma}\right\}~~ \mathrm {subject~to} ~~ \mathbf{H}\boldsymbol{\gamma}=\mathbf{0}, \label{EQ:minimization}$$ where $\mathbf{P}$ is the block diagonal penalty matrix satisfying that $\boldsymbol{\gamma}^{\T}\mathbf{P}\boldsymbol{\gamma}=\mathcal{E}_{\upsilon }(\mathbf{B}\boldsymbol{\gamma})$. To solve the constrained minimization problem (\[EQ:minimization\]), we first remove the constraint via QR decomposition of the transpose of the constrain matrix $\mathbf{H}$. Specifically, we write $$\label{QR4H} \mathbf{H}^{\T}=\mathbf{Q}\mathbf{R}=\left(\mathbf{Q}_1 ~\mathbf{Q}_2\right) \left( \begin{array}{c} \mathbf{R}_{1}\\ \mathbf{R}_{2} \end{array} \right),$$ where $\mathbf{Q}$ is an orthogonal matrix and $\mathbf{R}$ is an upper triangle matrix, the submatrix $\mathbf{Q}_1$ is the first $r$ columns of $\mathbf{Q}$, where $r$ is the rank of matrix $\mathbf{H}$, and $\mathbf{R}_2$ is a matrix of zeros. It is easy to see the following result. Let $\mathbf{Q}_1, \mathbf{Q}_2$ be submatrices as in . Let $\boldsymbol{\gamma} = \mathbf{Q}_2\boldsymbol{\theta}$ for a vector $\boldsymbol{\theta}$ of appropriate size. Then $\mathbf{H}\boldsymbol{\gamma} = \mathbf{0}$. On the other hand, if $\mathbf{H}\boldsymbol{\gamma} = \mathbf{0}$, then there exists a vector $\boldsymbol{\theta}$ such that $\boldsymbol{\gamma} = \mathbf{Q}_2\boldsymbol{\theta}$. By , we have $\mathbf{H}^{\T} = \mathbf{Q}_1 \mathbf{R}_{1}$ since $\mathbf{R}_2=\mathbf{0}$. That is, $\mathbf{H}= \mathbf{R}_{1}^{\T} \mathbf{Q}_1^{\T}$. Thus, $$\begin{aligned} \mathbf{H} \boldsymbol{\gamma}=& \mathbf{H} \mathbf{Q}_2\boldsymbol{\theta}= \mathbf{R}_{1}^{\T} \mathbf{Q}_1^{\T} \mathbf{Q}_2\boldsymbol{\theta} = \mathbf{0}\end{aligned}$$ since $\mathbf{Q}_1^{\T} \mathbf{Q}_2=\mathbf{0}$. On the other hand, if $$\mathbf{0}=\mathbf{H} \boldsymbol{\gamma} = \mathbf{R}_{1}^{\T} \mathbf{Q}_1^{\T}\boldsymbol{\gamma},$$ we have $\mathbf{Q}_1^{\T}\boldsymbol{\gamma}=0$ since $\mathbf{R}_1$ is invertible. Thus, $\boldsymbol{\gamma}$ is in the perpendicular subspace of the space spanned by the columns of $\mathbf{Q}_1$. That is, $\boldsymbol{\gamma}$ is in the space spanned by the columns of $\mathbf{Q}_2$. Thus, there exists a vector $\boldsymbol{\theta}$ such that $\boldsymbol{\gamma} = \mathbf{Q}_2\boldsymbol{\theta}$. These complete the proof. $\square $ The problem (\[EQ:minimization\]), is now converted to a conventional penalized regression problem without any constraints: $$\min_{\boldsymbol{\beta},\boldsymbol{\theta}} \left\{\|\mathbf{Y}-\mathbf{Z}\boldsymbol{\beta}-\mathbf{B}\mathbf{Q}_{2}\boldsymbol{\theta}\|^{2}+\lambda(\mathbf{Q}_{2}\boldsymbol{\theta})^{\T}\mathbf{P}(\mathbf{Q}_{2}\boldsymbol{\theta})\right\}. \label{EQ:minimization1}$$ For a fixed penalty parameter $\lambda$, we have $$\left( \begin{array}{c} \widehat{\boldsymbol{\beta}}\\ \widehat{\boldsymbol{\theta}} \end{array} \right) =\left\{\left(\begin{array}{cc} \mathbf{Z}^{\T}\mathbf{Z} &\mathbf{Z}^{\T}\mathbf{B}\mathbf{Q}_{2}\\ \mathbf{Q}_{2}^{\T}\mathbf{B}^{\T}\mathbf{Z} &\mathbf{Q}_{2}^{\T}\mathbf{B}^{\T}\mathbf{B}\mathbf{Q}_{2} \end{array}\right)+\lambda \left( \begin{array}{cc} \mathbf{0}&\\ & \mathbf{Q}_{2}^{\T}\mathbf{P}\mathbf{Q}_{2} \end{array} \right)\right\}^{-1} \left( \begin{array}{c} \mathbf{Z}^{\T}\mathbf{Y}\\ \mathbf{Q}_{2}^{\T}\mathbf{B}^{\T}\mathbf{Y} \end{array} \right).$$ Letting $$\mathbf{V}=\left( \begin{array}{cc} \mathbf{V}_{11} & \mathbf{V}_{12} \\ \mathbf{V}_{21} & \mathbf{V}_{22} \end{array}\right) =\left( \begin{array}{cc} \mathbf{Z}^{\T}\mathbf{Z} & \mathbf{Z}^{\T}\mathbf{B}\mathbf{Q}_{2}\\ \mathbf{Q}_{2}^{\T} \mathbf{B}^{\T} \mathbf{Z} & \mathbf{Q}_{2}^{\T}(\mathbf{B}^{\T} \mathbf{B}+\lambda\mathbf{P})\mathbf{Q}_{2} \end{array} \right), \label{DEF:V}$$ we have $$\left( \begin{array}{c} \widehat{\boldsymbol{\beta}}\\ \widehat{\boldsymbol{\theta}} \end{array} \right) =\mathbf{V}^{-1} \left(\begin{array}{c} \mathbf{Z}^{\T}\mathbf{Y}\\ \mathbf{Q}_{2}^{\T}\mathbf{B}^{\T}\mathbf{Y} \end{array}\right).$$ Next let us write $$\mathbf{V}^{-1}\equiv\mathbf{U}=\left( \begin{array}{cc} \mathbf{U}_{11} & \mathbf{U}_{12} \\ \mathbf{U}_{21} & \mathbf{U}_{22} \end{array}\right) =\left( \begin{array}{cc} \mathbf{U}_{11} & -\mathbf{U}_{11} \mathbf{V}_{12} \mathbf{V}_{22}^{-1} \\ - \mathbf{U}_{22} \mathbf{V}_{21} \mathbf{V}_{11}^{-1} & \mathbf{U}_{22} \end{array} \right) , \label{DEF:V-inverse}$$ where $$\begin{aligned} \mathbf{U}_{11}^{-1}&=&\mathbf{V}_{11}-\mathbf{V}_{12}\mathbf{V}_{22}^{-1}\mathbf{V}_{21} =\mathbf{Z}^{\T}\left[\mathbf{I}-\mathbf{B}\mathbf{Q}_2\{\mathbf{Q}_2^{\T}(\mathbf{B}^{\T} \mathbf{B} +\lambda\mathbf{P})\mathbf{Q}_2\}^{-1} \mathbf{Q}_2^{\T}\mathbf{B}^{\T}\right]\mathbf{Z},\label{DEF:U11}\\ \mathbf{U}_{22}^{-1}&=&\mathbf{V}_{22}-\mathbf{V}_{21}\mathbf{V}_{11}^{-1}\mathbf{V}_{12} =\mathbf{Q}_{2}^{\T}\left[\mathbf{B}^{\T} \left\{\mathbf{I}- \mathbf{Z} (\mathbf{Z}^{\T}\mathbf{Z})^{-1} \mathbf{Z}^{\T}\right\}\mathbf{B}+\lambda\mathbf{P} \right]\mathbf{Q}_{2}. \label{DEF:U22}\end{aligned}$$ Then the minimizers of (\[DEF:V\]) can be given precisely as follows: $$\begin{aligned} \widehat{\boldsymbol{\beta}}&\!\!\!\!=\!\!\!\!& \mathbf{U}_{11} \mathbf{Z}^{\T}\left(\mathbf{I}-\mathbf{B}\mathbf{Q}_2\mathbf{V}_{22}^{-1} \mathbf{Q}_2^{\T}\mathbf{B}^{\T}\right)\mathbf{Y} \!\!=\!\!\mathbf{U}_{11} \mathbf{Z}^{\T}\left\{\mathbf{I}-\mathbf{B}\mathbf{Q}_2\{\mathbf{Q}_2^{\T}(\mathbf{B}^{\T} \mathbf{B} +\lambda\mathbf{P})\mathbf{Q}_2\}^{-1} \mathbf{Q}_2^{\T}\mathbf{B}^{\T}\right\}\mathbf{Y},\\ \widehat{\boldsymbol{\theta}}&\!\!\!\!=\!\!\!\!&\mathbf{U}_{22}\mathbf{Q}_{2}^{\T}\mathbf{B}^{\T} \left(\mathbf{I} -\mathbf{Z} \mathbf{V}_{11}^{-1}\mathbf{Z}^{\T}\right)\mathbf{Y} \!\!=\!\!\mathbf{U}_{22}\mathbf{Q}_{2}^{\T}\mathbf{B}^{\T} \left\{\mathbf{I} -\mathbf{Z} (\mathbf{Z}^{\T}\mathbf{Z})^{-1}\mathbf{Z}^{\T}\right\}\mathbf{Y}.\end{aligned}$$ Therefore, one obtains the estimators for $\boldsymbol{\gamma}$ and $g(\cdot)$, respectively: $$\widehat{\boldsymbol{\gamma}}=\mathbf{Q}_{2}\widehat{\boldsymbol{\theta}} =\mathbf{Q}_{2}\mathbf{U}_{22}\mathbf{Q}_{2}^{\T}\mathbf{B}^{\T} \left\{\mathbf{I} -\mathbf{Z} (\mathbf{Z}^{\T}\mathbf{Z})^{-1}\mathbf{Z}^{\T}\right\}\mathbf{Y},$$ $$\widehat{g}(\mathbf{x})=\mathbf{B}(\mathbf{x})^{\T}\widehat{\boldsymbol{\gamma}}=\sum_{\xi \in \mathcal{K}} B_{\xi}( \mathbf{x})\widehat{\gamma}_{\xi}. \label{EQN:g_hat}$$ The fitted values at the $n$ data points are $$\widehat{\mathbf{Y}}=\mathbf{Z}\widehat{\boldsymbol{\beta}}+\mathbf{B} \widehat{\boldsymbol{\gamma}} =\mathbf{S}(\lambda)\mathbf{Y},$$ where the smoothing or hat matrix is $$\mathbf{S}(\lambda)=\mathbf{Z}\mathbf{U}_{11} \mathbf{Z}^{\T}\left(\mathbf{I}-\mathbf{B}\mathbf{Q}_2\mathbf{V}_{22}^{-1} \mathbf{Q}_2^{\T}\mathbf{B}^{\T} \right)+\mathbf{B}\mathbf{Q}_{2} \mathbf{U}_{22}\mathbf{Q}_{2}^{\T}\mathbf{B}^{\T} \left(\mathbf{I} -\mathbf{Z} \mathbf{V}_{11}^{-1}\mathbf{Z}^{\T}\right).$$ In nonparametric regression, the trace $\mathrm{tr}(\mathbf{S}(\lambda))$ of smoothing matrix $\mathbf{S}(\lambda)$ is often called the degrees of freedom of the model fit. It has the rough interpretation as the equivalent number of parameters and can be thought as a generalization of the definition in linear regression. Finally, we can estimate the variance of the error term, $\sigma^{2}$ by $$\label{EQ:sigma2_hat} \hat{\sigma}^{2}=\frac{\|\mathbf{Y}-\widehat{\mathbf{Y}}\|^{2}}{n-\mathrm{tr}(\mathbf{S}(\lambda))}.$$ 0.05in **2.4. Choosing the Triangulation** 0.1in Triangulation has been extensively investigated in the past few decades, and various packages have been developed. For example, one can use the "Delaunay" algorithm to find a triangulation; see MATLAB program *delaunay.m* or MATHEMATICA function *DelaunayTriangulation*. "Triangle" (Shewchuk, 1996) is also widely used in many applications, and one can download it for free from *http://www.cs.cmu.edu/ quake/triangle.html*. It is a C$++$ program for two-dimensional mesh generation and construction of Delaunay triangulations. "DistMesh" is another method to generate unstructured triangular and tetrahedral meshes; see the *DistMesh* generator on *http://persson.berkeley.edu/distmesh/*. A detailed description of the program is provided by Persson and Strang (2004). We used our own triangulation code in simulation studies and real data analysis below. As is usual with the one-dimensional (1-D) penalized least squares (PLS) splines, the number of knots is not important given that it is above some minimum depending upon the degree of the smoothness; see Li and Ruppert (2008). For bivariate PLS splines, Lai and Wang (2013) also observed that the number of triangles $N$ is not very critical, provided $N$ is larger than some threshold. In fact, one of the main advantages of using PLS splines over discrete least squares (DLS) splines is the flexibility of choosing knots in the 1-D setting and choosing triangles in the 2-D setting. For DLS splines, one has to have large enough sample according to the requirement of the degree of splines on each subinterval in the 1-D case or each triangle in the 2-D case to guarantee that a solution can be found. However, there is no such requirement for PLS splines. When the smoothness $r \geq 1$, the only requirement for bivariate PLS splines is that there is at least one triangle containing three points which are not in one line (Lai, 2008). Also, PLS splines perform similar to DLS splines as long as the penalty parameter $\lambda$ is very small. So in summary, the proposed bivariate PLS splines are very flexible and convenient for data fitting, even for smoothing sparse and unevenly sampled data. In practice, we recommend that the user first constructs a polygon domain $\Omega$ containing all the design points of the data and makes a simple triangulation $\triangle_{0}$ of $\Omega$ by hand or computer, then refines $\triangle_{0}$ several times to have a triangulation of desired size. 0.05in **2.5. Penalty Parameter Selection** 0.1in Selecting a suitable value of smoothing parameter $\lambda$ is critical to good model fitting. A large value of $\lambda$ enforces a smoother fitted function with potentially larger fitting errors, while a small value yields a rougher fitted function and potentially smaller fitting errors. Since the in-sample fitting errors can not gauge the prediction property of the fitted function, one should target a criterion function that mimics the out-of-sample performance of the fitted model. The generalized cross-validation (GCV) (Craven and Wahba, 1979; Wahba, 1990) is such a criterion and is widely used for choosing the penalty parameter. We choose the smoothing parameter $\lambda$ by minimizing the following generalized cross-validation (GCV) criterion $$\mathrm{GCV}(\lambda)=\frac{n\|\mathbf{Y}-\mathbf{S}(\lambda)\mathbf{Y}\|^2}{\{n-\mathrm{tr}(\mathbf{S}(\lambda))\}^2},$$ over a grid of values of $\lambda$. We use the 10-point grid where the values of $\log_{10}(\lambda)$ are equally spaced between $-6$ and $7$. \[sec:asymptotics\] .12in **3. IAsymptotic Results** 0.10in This section studies the asymptotic properties for the proposed estimators. To discuss these properties, we first introduce some notation. For any function $f$ over the closure of domain $\Omega$, denote $\Vert f\Vert _{\infty} =\sup_{\mathbf{x}\in \Omega} |f(\mathbf{x})|$ the supremum norm of function $f$ and $|f|_{\upsilon,\infty}= \max_{i+j=\upsilon}\Vert D_{x_{1}}^{i}D_{x_{2}}^{j}f(\mathbf{x})\Vert _{\infty}$ the maximum norms of all the $\upsilon $th order derivatives of $f$ over $\Omega$. Let $$W^{\ell,\infty }(\Omega)=\left\{f \mathrm{~on~} \Omega:|f|_{k,\infty}<\infty, 0\le k\le \ell \right\} \label{DEF:Sobolev}$$ be the standard Sobolev space. For any $j=1,\ldots, p$, let $z_{j}$ be the coordinate mapping that maps $\mathbf{z}$ to its $j$-th component so that $z_{j}(\mathbf{Z}_{i})=Z_{ij}$, and let $$h_j=\mathrm{argmin}_{h\in L^{2}}\|z_{j}-h\|_{L^{2}}^{2}=\mathrm{argmin}_{h\in L^{2}}E\{(Z_{ij}-h(\mathbf{X}_{i}) )^2\} \label{EQ:h_j}$$ be the orthogonal projection of $z_{j}$ onto $L^{2}$. Before we state the results, we make the following assumptions: - The random variables $Z_{ij}$ are bounded, uniformly in $i=1, \ldots, n$, $j =1,\ldots, p$. - The eigenvalues of $E\left\{\left.\left(\begin{array}{cc} 1 & \mathbf{Z}_{i}^{\T} \end{array}\right)^{\T} \left(\begin{array}{cc} 1& \mathbf{Z}_{i}^{\T} \end{array}\right)\right| \mathbf{X}_{i}\right\}$ are bounded away from 0. - The noise $\epsilon$ satisfies that $\lim_{\eta \rightarrow \infty}E\left[ \epsilon^{2}I(\epsilon>\eta)\right] =0$. Assumptions (A1)–(A3) are typical in semiparametric smoothing literature, see for instance Huang, Zhang and Zhou (2007) and Wang, et al (2011). The purpose of Assumption (A2) is to ensure that the vector $(1,\mathbf{Z}_{i}^{\T})$ is not multicolinear. We next introduce some assumptions on the properties of the true bivariate function in model (\[model\]) and the bivariate spline space introduced in Section 2.2. - The bivariate functions $h_{j}(\cdot)$, $j=1,\ldots, p$, and the true function in model (\[model\]) $g(\cdot) \in W^{\ell+1 ,\infty}(\Omega)$ in (\[DEF:Sobolev\]) for an integer $\ell \ge 2$. - For every $s\in \mathbb{S}$ and every $\tau \in \triangle$, there exists a positive constant $F_1$, independent of $s$ and $\tau$, such that $$\begin{aligned} F_1 \|s \|_{\infty,\tau}\le \left(\sum_{\mathbf{X}_i\in \tau, \ i=1, \cdots, n} s\left(\mathbf{X}_i\right)^2\right)^{1/2}, \quad \mbox{ for all } \tau \in \triangle. \label{EQ:F1}\end{aligned}$$ - Let $F_2$ be the largest among the numbers of observations in triangles $\tau\in \triangle$ in the sense that $$\begin{aligned} \left(\sum_{\mathbf{X}_i\in \tau, \ i=1, \cdots, n} s\left(\mathbf{X}_i\right)^2\right)^{1/2}\le F_2 \|s\|_{\infty,\tau}, \quad \mbox{ for all } \tau \in \triangle, \label{EQ:F2}\end{aligned}$$ where $\|s\|_{\infty, \tau}$ denotes the supremum norm of $s$ over triangle $\tau$. The constants $F_1$ and $F_2$ defined in (\[EQ:F1\]) and (\[EQ:F2\]) satisfy $F_2/F_1=O(1)$. - The number $N$ of the triangles and the sample size $n$ satisfy that $N=Cn^{\gamma}$ for some constant $C>0$ and $1/(\ell+1)\leq \gamma \leq 1/3$. - The penalized parameter $\lambda$ satisfies $\lambda=o(n^{1/2}N^{-1})$. Condition (C1) describes the requirement for the true bivariate function as usually used in the literature of nonparametric or semiparametric estimation. Condition (C2) ensures the existence of a least squares spline. Although one can get a decent penalized least squares spline fitting with $F_1=0$ for some triangles, we need (C2) to study the convergence of bivariate penalized least squares splines. Condition (C3) suggests that we should not put too many observations in one triangle. Condition (C4) requires that the number of triangles is above some minimum depending upon the degree of the spline, which is similar to the requirement of Li and Ruppert (2008) in the univariate case. It also ensures the asymptotic equivalence of the theoretical and empirical inner products/norms defined at the beginning of Section \[sec:asymptotics\]. Condition (C5) is required to reduce the bias of the bivariate spline approximation through "under smoothing" and "choosing smaller $\lambda$". To avoid confusion, in the following let $\boldsymbol{\beta}_0$ and $g_{0}$ be the true parameter value and function in model (\[model\]). The following theorem states that the rate convergence of $\widehat{\boldsymbol{\beta}}$ is root-$n$ and $\widehat{\boldsymbol{\beta}}$ is asymptotically normal. \[THM:beta-normality\] Suppose Assumptions (A1)-(A3), (C1)-(C5) hold, then the estimator $\widehat{\boldsymbol{\beta}}$ is asymptotically normal, that is, $$(n\boldsymbol{\Sigma})^{1/2}(\widehat{\boldsymbol{\beta}}-\boldsymbol{\beta}_{0}) \rightarrow N(\mathbf{0}, \mathbf{I}),$$ where $\mathbf{I}$ is a $p\times p$ identity matrix, $$\boldsymbol{\Sigma}=\sigma^{-2}E\{(\mathbf{Z}_{i}-\widetilde{\mathbf{Z}}_{i}) (\mathbf{Z}_{i}-\widetilde{\mathbf{Z}}_{i})^{\T}\} \label{DEF:Sigma}$$ with $\widetilde{\mathbf{Z}}_{i}=\left\{h_{1}(\mathbf{X}_{i}), \ldots, h_{p}(\mathbf{X}_{i})\right\}^{\T}$, for $h_{j}(\cdot)$ defined in (\[EQ:h\_j\]), $j=1,\ldots,p$. In addition, $\boldsymbol{\Sigma}$ can be consistently estimated by $$\boldsymbol{\Sigma}_{n}=\frac{1}{n\widehat{\sigma}^{2}}\sum_{i=1}^{n} (\mathbf{Z}_{i}-\widehat{\mathbf{Z}}_{i}) (\mathbf{Z}_{i}-\widehat{\mathbf{Z}}_{i})^{\T} =\frac{1}{n\widehat{\sigma}^{2}} (\mathbf{Z}-\widehat{\mathbf{Z}})^{\T}(\mathbf{Z}-\widehat{\mathbf{Z}}). \label{DEF:Sigma_n}$$ where $\widehat{\mathbf{Z}}_{i}$ is the $i$-th column of $\widehat{\mathbf{Z}}^{\T}=\mathbf{Z}^{\T}\mathbf{B}\mathbf{Q}_2\mathbf{V}_{22}^{-1} \mathbf{Q}_2^{\T}\mathbf{B}^{\T}$ and $\widehat{\sigma}^{2}$ is given by (\[EQ:sigma2\_hat\]). The results in Theorem \[THM:beta-normality\] enable us to construct confidence intervals for the parameters. The next theorem provides the global convergence of the nonparametric estimator $\widehat{g}(\cdot)$. \[THM:g-convergence\] Suppose Assumptions (A1)-(A3), (C1)-(C5) hold, then the bivariate penalized estimator $\widehat{g}(\cdot)$ in ( \[EQN:g\_hat\]) is consistent with the true function $g_{0}$, and satisfies that $$\Vert \widehat{g}-g_{0}\Vert _{L^2}=O_{P}\left( \frac{\lambda }{n\left| \triangle \right| ^{3}}|g_{0}|_{2,\infty} +\left(1+\frac{\lambda }{n\left| \triangle \right| ^{5}}\right) \frac{F_{2}}{F_{1}}|\triangle |^{\ell +1}|g_{0}|_{\ell,\infty}+\frac{1}{\sqrt{n}|\triangle|} \right).$$ The proofs of the above two theorems are given in Appendix. We notice that the rate of convergence given in Theorem \[THM:g-convergence\] is the same as those for nonparametric spline regression without including the covariate information obtained in Lai and Wang (2013). .12in **4. Simulation** \[sec:simulation\] 0.10in In this section, we carry out two numerical studies to assess the performance of our proposed method. We compare the performance of bivariate penalized splines over triangulations (BPST) with filtered kriging (KRIG), thin plate splines (TPS), soap film smoothing (SOAP) in Wood, Bravington and Hedley (2008), linear finite elements (LFE) and quadratic finite elements (QFE) in Sangalli, Ramsay and Ramsay (2013). *Example 1.* In this example, we consider a modified horseshoe domain with the surface test function $g(\cdot)$ used by Wood, Bravington and Hedley (2008) and Sangalli, Ramsay and Ramsay (2013). In particular, for $201\times 501$ grid points over the domain, we simulate data as follows, the response variable $Y$ is generated from the following PLM: $$Y=\beta_{1}Z_{1}+\beta_{2}Z_{2}+g(X_1,X_2)+\epsilon.$$ The random error, $\epsilon$, is generated from an $N(0,\sigma_{\epsilon}^{2})$ distribution with $\sigma_{\epsilon}=0.5$. In addition, we set the parameters as $\beta_{1}=-1$, $\beta_{2}=1$. For the design of the explanatory variables, $Z_{1}$ and $Z_{2}$, two scenarios are considered based on the relationship between the location variables $(X_1,X_2)$ and covariates $(Z_1, Z_2)$. Under both scenarios, $Z_{1}\sim uniform[-1,1]$. On the other hand, the variable $Z_{2}=\cos[\pi (\rho(X_{1}^{2}+X_{2}^{2})+(1-\rho)U)]$ where $U\sim uniform[-1,1]$ and is independent from $(X_1,X_2)$ as well as $Z_{1}$. We consider both independent design: $\rho=0.0$ and dependent design: $\rho=0.7$ in this example. Under both scenarios, 100 Monte Carlo replicates are generated. For each replication, we randomly sample $n=200$ locations uniformly from the grid points inside the horseshoes domain. Figure \[FIG:eg1\_1\] (a) and (b) show the surface and the contour map of the true function $g(\cdot)$, respectively. Figure \[FIG:eg1\_1\] (c) demonstrates the sampled location points of replicate 1, and Figure\[FIG:eg1\_1\] (d) and (e) illustrates two different triangulations used in the BPST method. In the first triangulation ($\triangle_1$), we use $91$ triangles and $74$ vertices, while in the second one ($\triangle_2$) we use $109$ triangles and $95$ vertices. \[Figure \[FIG:eg1\_1\] about here\] To implement the TPS and KRIG methods, we use the R package *fields* under the standard implementation setting of Furrer, Nychka and Sainand (2011). For KRIG, we try different covariance structures, and we choose the Matérn covariance with smoothness parameter $\nu=1$, which gives the best prediction. For the SOAP method, we implement it by using R package *mgcv* (Wood, Bravington and Hedley, 2008) with $32$ interior knots. In addition, a rank 39 (40-knot) cyclic penalized cubic regression spline is used as the boundary curve. For all the methods requiring a smoothing or roughness parameter, GCV is used to choose the values of the parameter. To see the accuracy of the estimators, we compute the root mean squared error (RMSE) for each of the components based on $100$ Monte carlo samples. Table \[TAB:eg1\_1\] shows the RMSEs of the estimate of the parameters $\beta_{1}$, $\beta_{2}$, $\sigma_{\varepsilon}$ as well as the average mean squared prediction error (MSPE) of the nonlinear function $g(\cdot)$ over all the grid points in the horseshoes shaped domain. From Table \[TAB:eg1\_1\], one sees that methods SOAP and BPST give very comparable estimates of the parameters $\beta_{1}$ and $\beta_{2}$, respectively, while BPST method produces the best prediction of the nonlinear function $g(\cdot$), regardless of the choice of triangulation. \[Table \[TAB:eg1\_1\] about here\] Figure \[FIG:eg1\_2\] shows the estimated functions via different methods for the first replicate. For the test function on Figure \[FIG:eg1\_2\], the BPST estimate looks visually better than the other four estimates. In addition, one sees there is a "leakage effect" in KRIG and TPS estimates and the SOAP, LFE, QFE and BPST methods improve the model fitting of KRIG and TPS. The poor performance of KRIG and TPS is because they do not take the complex boundary into any account and smooth across the gap inappropriately. In addition, from Figure \[FIG:eg1\_2\], one also sees that the BPST estimators based on $\triangle_1$ and $\triangle_2$ are very similar, which agrees our findings for penalized splines that the number of triangles is not very critical for the fitting as long as it is large enough. \[Figure \[FIG:eg1\_2\] about here\] Next we test the accuracy of the standard error formula in (\[DEF:Sigma\_n\]) for $\hat{\beta}_{1}$ and $\hat{\beta}_{2}$, and the results are listed in Table \[TAB:eg1\_2\]. The standard deviations of the estimated parameters are computed based on $100$ replications, which can be regarded as the true standard errors (column labeled "$\mathrm{SE}_{\mathrm{mc}}$") and compared with the mean and median of the $100$ estimated standard errors calculated using (\[DEF:Sigma\_n\]) (columns labeled "$\mathrm{SE}_{\mathrm{mean}}$" and "$\mathrm{SE}_{\mathrm{median}}$", respectively). The column labeled "$\mathrm{SE}_{\mathrm{mad}}$" is the interquartile range of the $100$ estimated standard errors divided by $1.349$, which is a robust estimate of the standard deviation. From Table \[TAB:eg1\_2\] one observes that the averages or medians of the standard errors calculated using the formula are very close to the true standard deviations, which confirms the accuracy of the proposed standard error formula. \[Table \[TAB:eg1\_2\] about here\] Finally, for computing time, it takes about $60$ seconds to compute BPST in one run of simulation with even $2000$ observations over $319$ triangles under the computing environment of x64 PC with Intel Dual Core i5. Overall, the proposed algorithm is fast to compute. *Example 2.* In this example, we consider a rectangular domain, $[0,1]^2$, where there is no irregular shape or complex boundaries problem. In this case, classical methods for spatial data analysis, such as KRIG and TPS, will not encounter any difficulty. We obtain the true signal and noisy observation for each coordinate pair lying on a $101\times 101$-grid over $[0,1]^2$ using the following model: $$Y=\mathbf{Z}^{\T}\boldsymbol{\beta}+g(X_{1},X_{2})+\xi(X_{1},X_{2})+\epsilon,$$ where $\boldsymbol{\beta} = (-1,1)^{\T}$ and $g(x_1,x_2)=5\sin (2\pi(x_1^2+x_2^2))$. The random error, $\epsilon$, is generated from an $N(0,\sigma_{\epsilon}^{2})$ distribution with $\sigma_{\epsilon}=0.5$, and the process $\xi(\cdot)$ is generated from a stationary gaussian random field with the Matérn(1,1) covariance structure. The components of $\mathbf{Z}$ and $\epsilon$ are standard normal, and $\mathbf{Z}$, $\xi$ and $\epsilon$ are independent of each other. Similar to Example 1, we simulate $Z_{1}\sim uniform[-1,1]$, and $Z_{2}=cos[\pi (\rho(X_{1}^{2}+X_{2}^{2})+(1-\rho)U)]$, where $\rho=0.0~\mathrm{or}~0.7,~U\sim uniform[-1,1]$ and is independent from $(X_{1},X_{2})$ and $Z_{1}$. Next we take $100$ Monte Carlo random samples of size $n=200$ from the $101\times101$ points. Figure \[FIG:eg2\_1\] (a) and (b) display the true sinusoidal surface and the contour map, respectively. \[FIG:eg2\_1\] (c) represents the Matérn structure. We use the triangulation in Figure \[FIG:eg2\_1\] (e) and (f), and there are 8 triangles and 9 vertices as well as 32 triangles and 25 vertices, respectively. In addition, the points in Figure \[FIG:eg2\_1\] (d) demonstrate the sampled location points of replicate 1. \[Figure \[FIG:eg2\_1\] about here\] Similar to the study in Example 4.1, we also compare the proposed BPST estimator with estimators from the KRIG, TPS, SOAP, LFE and QFE methods, which are implemented in the same way as Example 4.1. Specifically, for KRIG, we choose the true Matérn covariance with smoothness parameter $\nu=1$. To see the accuracy of the estimators, we compute the RMSEs of the coefficient estimators and the estimator of $\sigma_{\epsilon}$. To see the overall prediction accuracy, we make prediction over the $101 \times 101$ grid points on the domain for each replication using different methods, and compare the predicted values with the true observations of $Y$ at these grid points, and we report the average mean squared prediction errors (MSPE) over all replications. All the results are summarized in Table \[TAB:eg2\_1\]. As expected, KRIG works pretty well since the true covariance structure is used in the KRIG fitting. When $\rho=0$, TPS, KRIG and BPST all perform very well. When $\rho=0.7$, KRIG and BPST are the best among all the estimators, and both of them outperform the rest of the estimators. In both scenarios, the differences between BPST and KRIG are almost unnoticeable. One also notices that, compared with the FEM estimators (LEM and QEM), our BPST estimator shows much better performance in terms of both estimation and prediction, because BPST provides a more flexible and easier construction of splines with piecewise polynomials of various degrees and smoothness than the FEM method. Finally as pointed out in Wood, Bravington and Hedley (2008), the standard (linear) FEM method requires a very fine triangulation in order to reach certain approximation power, however, BPST doesn't need such a strict fineness requirement as it uses piecewise polynomials of degree $\ge 5$ yielding an higher order approximation power. In fact, we have used 64 times more triangles in the FEM than that for the BPST in our simulation. That is, the BPST is computationally more efficient than the FEM to approximate smooth functions. \[Table \[TAB:eg2\_1\] about here\] Table \[TAB:eg2\_2\] lists the accuracy results of the standard error formula in (\[DEF:Sigma\_n\]) for $\hat{\beta}_{1}$ and $\hat{\beta}_{2}$ using BPST. From Table \[TAB:eg2\_2\], one sees that the estimated standard errors based on sample size $n = 200$ are very accurate. \[Table \[TAB:eg2\_2\] about here\] .12in **5. Application to California House Value Data** \[sec:application\] 0.10in In this section we apply the proposed method to analyze the California house value data from the StatLib repository. The data appeared in Pace and Barry (1997). The spatial data consists of information of all the block groups in California defined by centroid of census enumeration areas. In the data set, a block group on average includes $1425.5$ individuals living in a geographically compact area and there are $20,640$ blocks in the final data. In this paper, we study how different features and factors influence the real estate property prices. The response variable is the median house value (Value). The investigated factors include the median income (MedInc), median house age (Age), the average number of bedrooms (AveBedrms), housing density as reflected by the number of households (Hhd) in each block, and the average occupancy in each household (AveOccup). It is obvious that the location of a house is very crucial for making an accurate prediction, so we also include the latitude (Latitude) and longitude (Longitude) of the block. We model the median house value using the following PLM: $$\begin{aligned} \!\!\!\!\log(\textrm{Value})&\!\!\!\!=\!\!\!\!&\beta_{0}+\beta_{1}\textrm{MedInc}+\beta_{2}\log(\textrm{Age}) + \beta_3 \log(\textrm{AveBedrms}) \notag \\ &&\!\!\!\! +\beta_4 \log(\textrm{AveOccup}) +\beta_5 \log({\textrm{Hhd}})+g(\textrm{Latitude, Longitude}). \label{app:plm}\end{aligned}$$ To fit model (\[app:plm\]), we use six different methods: KRIG, TPS, SOAP, LFE, QFE, and BPST. For comparison, we also consider the purely linear model without using the spatial information: $$\begin{aligned} \log(\textrm{Value})&\!\!\!=\!\!\!&\beta_{0}+\beta_{1}\textrm{MedInc}+\beta_{2}\log(\textrm{Age}) +\beta_3 \log(\textrm{AveBedrms})\notag \\ &&\!\!\! +\beta_4 \log(\textrm{AveOccup})+\beta_5 \log({\textrm{Hhd}}), \label{app:lm}\end{aligned}$$ and fit it using the ordinary linear least squares (OLS) method. In the following, we report our estimation results for the linear and nonlinear components in model (\[app:plm\]) by using the OLS, KRIG, TPS, SOAP, LFE, and QFE methods. Table \[TAB:APPest\] summarizes the coefficient estimation results of all procedures, where the BPST confidence intervals are constructed based on the standard error (s.e.) calculated using (\[DEF:Sigma\_n\]). Note that the coefficient on "$\log(\textrm{Age})$" is positive $0.165$ ($s.e.=0.006$) in the fitted linear regression model (\[app:lm\]) via OLS, which is obviously against the common sense in household real estate market. Using the PLM (\[app:plm\]) with the spatial coordinates, we obtain negative coefficients for "$\log(\textrm{Age})$" regardless which spatial based method is employed. The coefficient for the variable "$\log(\textrm{AveBedrms})$" is $-0.037$ with a standard error $0.018$ using OLS. In contrast, all the semiparametric methods unanimously suggest that the coefficient for "$\log(\textrm{AveBedrms})$" is positive in the PLM in (\[app:plm\]) after the location effect is controlled. In addition, the coefficient on "$\log(\textrm{Hhd})$" is positive $0.088$ and very significant ($s.e.=0.004$) in the OLS, but it is statistically insignificant when we apply the PLM (\[app:plm\]) to the data. In summary, compared with the OLS method, our results from PLM are more consistent with intuitions in the real estate market study. The above counter-intuitive phenomenon in OLS is perhaps due to the model misspecification of (\[app:lm\]) that completely ignores the location factor of the property. \[Table \[TAB:APPest\] about here\] Based on the median house values, we classify the houses in the dataset into six different groups: (1) less than 50K, (2) 50K–100K, (3) 100K–150K, (4)150K–200K, (5) 200K–300K, and (6) greater than 300K, and these groups are plotted in Figure \[FIG:cah\_vtrue\]. We plot the estimated median house values using different methods in Figure \[FIG:cah\_vpred\], where different colors are used to represent the value of houses as in Figure \[FIG:cah\_vtrue\]. All the plots in Figures \[FIG:cah\_vtrue\] and \[FIG:cah\_vpred\] show that expensive houses are clustered around the major cities and inland house values are lower than coastal house values. The OLS significantly underestimates the coastal enclaves of expensive houses and overestimates the house values in the central valley; see Figure \[FIG:cah\_vpred\] (a). In contrast, the PLM methods in Figure \[FIG:cah\_vpred\] (b)-(g) provide much more accurate estimates. \[Figures \[FIG:cah\_vtrue\], \[FIG:cah\_vpred\] about here\] Figure \[FIG:cah\_vdiff\] further demonstrates the differences between the estimated house values and the observed house values using seven different methods. As seen in Figure \[FIG:cah\_vdiff\], some methods, especially the OLS, have difficulties in estimating the house values in major metropolitan areas and the central valley. The proposed BPST method dramatically reduces the estimation errors in those areas. \[Figure \[FIG:cah\_vdiff\] about here\] To evaluate different methods, we also estimate the out-of-sample prediction errors of each method using the 5-fold cross validation. We randomly split all the observations into five roughly equal-sized parts. For each $k=1,\ldots,5$, we leave out part $k$, fit the model to the other four parts (combined), and then obtain predictions for the left-out $k$th part. Table \[TAB:APPmspe\] summarizes the mean squared prediction errors of the logarithm of the median house value based on different methods. As expected, incorporating the spatial information dramatically reduces the prediction errors, and the proposed BPST method provides the most accurate prediction on the median housing values among all the methods. \[Table \[TAB:APPmspe\] about here\] .10in **6. Concluding Remarks** \[sec:conclusion\] 0.05in In this paper, we have considered PLMs for analyzing spatial data. We introduce a framework of bivariate penalized splines over triangulations in the semiparametric estimation. Our work differs from existing works in four major aspects. First, the proposed estimator solves the problem of "leakage" across the complex domains where many conventional smoothing tools suffer from. The numerical results of the simulation show our method is very effective on both regular and irregular domains. Secondly, we provide new statistical theories for estimating the PLM for data distributed over complex spatial domains. It is shown that our estimates of both parametric part and non-parametric part of the model enjoy excellent asymptotic properties. In particular, we have shown that our estimates of the coefficients in the parametric part are asymptotically normal and derived the convergence rate of the nonparametric component under regularity conditions. We have also provided a standard error formula for the estimated parameters and our simulation studies show that the standard errors are estimated with good accuracy. The theoretical results provide measures of the effect of covariates after adjusting for the location effect. In addition, they give valuable insights into the accuracy of our estimate of the PLM and permit joint inference for the parameters. Thirdly, comparing with the conventional FEM, our method provides a more flexible and easier construction of splines with piecewise polynomials of various degrees and various smoothness. Finally, the proposed method greatly enhances the application of PLMs to spatial data analysis. We don't require the data to be evenly distributed or on regular-spaced grid. When we have regions of sparse data, PLS splines provides a more convenient tool for data fitting than DLS splines since the roughness penalty helps regularize the estimation. Our estimation is computationally fast and efficient since it can formulate a penalized regression problem using a QR decomposition. This paper leaves open the problem of how to choose a triangulation for a given data set. An optimal triangulation $\triangle$ enables us to achieve the best approximation of the given data using a spline space $\mathbb{S}(\triangle)$ of a fixed degree and fixed smoothness. We leave the problem for our future exploration. 0.05in <span>**Acknowledgment**</span> The first author's research was supported in part by National Science Foundation grants DMS-11-06816 and DMS-15-42332, and the third author's research was supported in part by National Science Foundation grant DMS-15-21537. 0.10in **Appendices** .05in **A.1. Preliminaries** In the following, we use $c$, $C$, $c_1$, $c_2$, $C_1$, $C_2$, etc. as generic constants, which may be different even in the same line. For functions $f_{1}$ and $f_{2}$ on $\Omega \times \mathbb{R}^{p}$, we define the empirical inner product and norm as $\left\langle f_{1},f_{2}\right\rangle_{n}=\frac{1}{n}\sum_{i=1}^{n} f_{1}(\mathbf{X}_{i},\mathbf{Z}_{i})f_{2}(\mathbf{X}_{i},\mathbf{Z}_{i})$ and $\left\| f_{1}\right\| _{n}^{2}=\left\langle f_{1},f_{1}\right\rangle_{n}$. If $f_{1}$ and $f_{2}$ are $L^2$-integrable, we define the theoretical inner product and theoretical $L^2$ norm as $\left\langle f_{1},f_{2}\right\rangle_{L^2} =E\left\{ f_{1}(\mathbf{X}_{i},\mathbf{Z}_{i})f_{2}(\mathbf{X}_{i},\mathbf{Z}_{i})\right\}$ and $\left\| f_{1}\right\| _{L^2}=\left\langle f_{1},f_{1}\right\rangle_{L^2}$. Furthermore, let $\left\| \cdot\right\| _{\mathcal{E}_{\upsilon }}$ be the norm introduced by the inner product $\left\langle \cdot, \cdot\right\rangle _{\mathcal{E}_{\upsilon }}$, where $$\left\langle g_{1},g_{2}\right\rangle _{\mathcal{E}_{\upsilon }}=\int_{\Omega}\left\{\sum_{i+j=\upsilon } \binom{\upsilon}{i} (D_{x_{1}}^{i}D_{x_{2}}^{j}g_{1})^{2}\right\}^{1/2}\left\{\sum_{i+j=\upsilon } \binom{\upsilon}{i} (D_{x_{1}}^{i}D_{x_{2}}^{j}g_{2})^{2}\right\}^{1/2}dx_{1}dx_{2}$$ for $g_{1}$ and $g_{2}$ on $\Omega$. <span>\[Lai and Schumaker (2007)\]</span> \[LEM:normequity\] Let $\{B_{\xi }\}_{\xi \in \mathcal{K}}$ be the Bernstein polynomial basis for spline space $\mathbb{S}$ with smoothness $r$, where $\mathcal{K}$ stands for an index set. Then there exist positive constants $c$, $C$ depending on the smoothness $r$ and the shape parameter $\delta$ such that $$c|\triangle |^{2}\sum_{\xi \in \mathcal{K}}|\gamma_{\xi }|^{2}\le \Vert \sum_{\xi \in \mathcal{K}}\gamma_{\xi }B_{\xi }\Vert _{L^2}^{2}\le C|\triangle |^{2}\sum_{\xi \in \mathcal{K}}|\gamma_{\xi}|^{2} \label{EQ:normequity}$$ for all $\gamma_{\xi },\xi \in \mathcal{K}$. With the above stability condition, Lai and Wang (2013) established the following uniform rate at which the empirical inner product approximates the theoretical inner product. <span>\[Lemma 2 of the Supplement of Lai and Wang (2013)\]</span> \[LEM:Rnorder\] Let $g_{1}=\sum_{\xi \in \mathcal{K}}c_{\xi }B_{\xi }$, $g_{2}=\sum_{\zeta \in \mathcal{K}}\widetilde{c}_{\zeta }B_{\zeta }$ be any spline functions in $\mathbb{S}$. Under Condition (C4), we have $$\sup\limits_{g_{1},g_{2}\in \mathbb{S}}\left| \frac{\left\langle g_{1},g_{2}\right\rangle_{n}-\left\langle g_{1},g_{2}\right\rangle _{L^2}}{\left\| g_{1}\right\| _{L^2}\left\| g_{2}\right\| _{L^2}}\right| =O_{P}\left\{(N\log n)^{1/2}/{n}^{1/2}\right\}.$$ <span>\[Corollary of Theorem 6 in Lai (2008)\]</span> \[LEM:appord\] Assume $g(\cdot)$ is in Sobolev space $W^{\ell +1,\infty }(\Omega )$. For bi-integer $(\alpha_{1},\alpha _{2})$ with $0\leq {\alpha _{1}}+{\alpha _{2}}\leq \upsilon $, there exists a unique spline fit $g^{*}(\cdot)\in \mathbb{S}$ such that $$\Vert D_{x_{1}}^{\alpha _{1}}D_{x_{2}}^{\alpha _{2}}\left( g-g^{*}\right) \Vert _{\infty}\le C\frac{F_{2}}{F_{1}}|\triangle |^{\ell +1-\alpha _{1}-\alpha _{2}}|g|_{\ell +1,\infty}, \label{eqGS}$$ where $C$ is an absolute constant depending on $r$ and $\delta$. For any smooth bivariate function $g(\cdot)$ and $\lambda>0$, define $$s_{\lambda,g}=\mathrm{argmin}_{s\in \mathbb{S}} \sum_{i=1}^{n}\{g(\mathbf{X}_{i})-s(\mathbf{X}_{i})\}^{2}+\lambda\mathcal{E}_{\upsilon}(s) \label{DEF:s_lambdag}$$ the penalized least squares splines of $g(\cdot)$. Then $s_{0,g}$ is the nonpenalized spline estimator of $g(\cdot)$. \[LEM:bias\] Assume $g(\cdot)$ is in Sobolev space $W^{\ell +1,\infty }(\Omega )$. For bi-integer $(\alpha_{1},\alpha _{2})$ with $0\leq {\alpha _{1}}+{\alpha _{2}}\leq \upsilon $, there exists an absolute constant $C$ depending on $r$ and $\delta$, such that with probability approaching 1, $$\Vert D_{x_{1}}^{\alpha _{1}}D_{x_{2}}^{\alpha _{2}}\left( g-s_{0,g}\right) \Vert _{\infty}\le C\frac{F_{2}}{F_{1}}|\triangle |^{\ell +1-\alpha _{1}-\alpha _{2}}|g|_{\ell +1,\infty}.$$ *Proof.* Note that $$\begin{aligned} \Vert D_{x_{1}}^{\alpha _{1}}D_{x_{2}}^{\alpha _{2}}\left( g-s_{0,g}\right) \Vert _{\infty} &\leq& \Vert D_{x_{1}}^{\alpha _{1}}D_{x_{2}}^{\alpha _{2}}\left( g-g^{*}\right) \Vert _{\infty} +\Vert D_{x_{1}}^{\alpha _{1}}D_{x_{2}}^{\alpha _{2}}\left( g^{*}-s_{0,g}\right) \Vert _{\infty}\\ &\leq& \Vert D_{x_{1}}^{\alpha _{1}}D_{x_{2}}^{\alpha _{2}}\left( g-g^{*}\right) \Vert _{\infty} +\Vert D_{x_{1}}^{\alpha _{1}}D_{x_{2}}^{\alpha _{2}}\left( s_{0,g^{*}-g}\right) \Vert _{\infty}.\end{aligned}$$ The desired result follows from Lemma \[LEM:appord\], the projection bound result in Golitschek and Schumaker (2002), and the differentiation properties of bivariate splines over triangulatins given in Lai and Schumaker (2007). $\square$ \[LEM:norder\] Suppose $g(\cdot)$ is in the Sobolev space $W^{\ell +1,\infty }(\Omega )$, and let $s_{\lambda,g}$ be its penalized spline estimator defined in (\[DEF:s\_lambdag\]). Under Conditions (C2) and (C3), $$\left\|g-s_{\lambda, g}\right\| _{n}= O_{P}\left\{\frac{F_{2}}{F_{1}}\left| \triangle \right| ^{\ell +1}\left|g\right| _{\ell+1,\infty}+\frac{\lambda }{n\left| \triangle \right| ^{2}}\left( \left|g\right| _{\upsilon,\infty}+\frac{F_{2}}{F_{1}}\left| \triangle \right| ^{\ell+1-\upsilon}\left|g\right| _{\ell +1,\infty}\right) \right\}.$$ *Proof.* Note that $s_{\lambda,g}$ is characterized by the orthogonality relations $$\begin{aligned} n\left\langle g-s_{\lambda, g},u\right\rangle_{n}=\lambda \left\langle s_{\lambda, g},u\right\rangle _{\mathcal{E}_{\upsilon }},\quad \textrm{for all }u\in \mathbb{S}, \label{EQ:pls}\end{aligned}$$ while $s_{0, g}$ is characterized by $$\begin{aligned} \left\langle g-s_{0, g},u\right\rangle_{n}=0,\quad \textrm{for all }u\in \mathbb{S}. \label{EQ:ls}\end{aligned}$$ By (\[EQ:pls\]) and (\[EQ:ls\]), $ n\left\langle s_{0, g}-s_{\lambda, g},u\right\rangle_{n}=\lambda \left\langle s_{\lambda,g},u\right\rangle _{\mathcal{E}_{\upsilon }}$, for all $u\in \mathbb{S}$. Replacing $u$ by $s_{0, g}-s_{\lambda,g}$ yields that $$\begin{aligned} n\left\| s_{0, g}-s_{\lambda, g}\right\| _{n}^{2}=\lambda \left\langle s_{\lambda, g},s_{0, g}-s_{\lambda, g}\right\rangle _{\mathcal{E}_{\upsilon }}. \label{EQ:pls-ls}\end{aligned}$$ Thus, by Cauchy-Schwarz inequality, $$n\left\| s_{0, g}-s_{\lambda, g}\right\| _{n}^{2}\leq \lambda \left\| s_{\lambda, g}\right\| _{\mathcal{E}_{\upsilon }}\left\| s_{0, g}-s_{\lambda, g}\right\| _{\mathcal{E}_{\upsilon }} \leq \lambda \left\| s_{\lambda, g}\right\| _{\mathcal{E}_{\upsilon }} \sup_{f\in \mathbb{S}}\left\{ \frac{\left\| f\right\| _{\mathcal{E}_{\upsilon }}}{\left\| f\right\| _{n}},\left\| f\right\| _{n}\neq 0\right\} \left\| s_{0, g}-s_{\lambda, g}\right\| _{n}.$$ Similarly, using (\[EQ:pls-ls\]), we have $$n\left\| s_{0, g}-s_{\lambda, g}\right\| _{n}^{2}=\lambda \left\{ \left\langle s_{\lambda, g},s_{0, g}\right\rangle _{\mathcal{E}_{\upsilon }}-\left\langle s_{\lambda, g},s_{\lambda, g}\right\rangle _{\mathcal{E}_{\upsilon }}\right\} \geq 0.$$ Therefore, by Cauchy-Schwarz inequality, $$\left\| s_{\lambda, g}\right\| _{\mathcal{E}_{\upsilon }}^{2}\leq \left\langle s_{\lambda ,g},s_{0, g}\right\rangle _{\mathcal{E}_{\upsilon }}\leq \left\| s_{\lambda, g}\right\| _{\mathcal{E}_{\upsilon }}\left\| s_{0, g}\right\| _{\mathcal{E}_{\upsilon}},$$ which implies that $\left\| s_{\lambda, g}\right\| _{\mathcal{E}_{\upsilon }}\leq \left\| s_{0, g}\right\| _{\mathcal{E}_{\upsilon }}$. Therefore, $$\left\| s_{0, g}-s_{\lambda, g}\right\| _{n}\leq n^{-1}\lambda \left\| s_{0, g}\right\| _{\mathcal{E}_{\upsilon }} \sup_{f\in \mathbb{S}}\left\{ \frac{\left\| f\right\| _{\mathcal{E}_{\upsilon }}}{\left\| f\right\| _{n}},\left\| f\right\| _{n}\neq 0\right\}.$$ By Lemma \[LEM:bias\], with probability approaching 1, $$\begin{aligned} \left\| s_{0, g}\right\| _{\mathcal{E}_{\upsilon }} &\leq &C_{1}A_\Omega\left\{ \left| g\right| _{\upsilon,\infty}+\sum_{\alpha _{1}+\alpha _{2}=\upsilon }\left\| D_{x_{1}}^{\alpha _{1}}D_{x_{2}}^{\alpha _{2}}\left( g - s_{0, g}\right) \right\| _{\infty}\right\} \notag\\ &\leq &C_{2}A_\Omega\left( \left| g\right| _{\upsilon,\infty}+\frac{F_{2}}{F_{1}}\left| \triangle \right| ^{\ell +1-\upsilon}\left| g\right| _{\ell +1,\infty}\right), \label{EQ:s_0g_Ev}\end{aligned}$$ where $A_\Omega$ denotes the area of $\Omega$. By Markov's inequality, for any $f\in \mathbb{S}$, $\left\|f\right\| _{\mathcal{E}_{\upsilon }}\leq C \left| \triangle \right| ^{-2}\left\|f\right\| _{L^2}$. Lemma (\[LEM:Rnorder\]) implies that $\sup_{f\in\mathcal{S}}\left\{ \left. \left\|f\right\| _{n}\right/ \left\|f\right\| _{L^2}\right\} \geq 1-O_{P}\left\{(N\log n)^{1/2}/{n}^{1/2}\right\}$. Thus, we have $$\sup_{f\in \mathbb{S}}\left\{ \frac{\left\| f\right\| _{\mathcal{E}_{\upsilon }}}{\left\| f\right\| _{n}},\left\| f\right\| _{n}\neq 0\right\} \leq C\left| \triangle \right| ^{-2}\left[ 1-O_{P}\left\{(N\log n)^{1/2}/{n}^{1/2}\right\} \right] ^{-1/2}=O_{P}\left( \left| \triangle \right| ^{-2}\right). \label{EQ:ev-vs-n}$$ Therefore, $$\left\| s_{0, g}-s_{\lambda, g}\right\| _{n} =O_P\left\{\frac{\lambda}{n\left| \triangle \right|^{2}} \left( \left| g\right| _{\upsilon,\infty}+\frac{F_{2}}{F_{1}}\left| \triangle \right| ^{\ell +1-\upsilon}\left| g\right| _{\ell +1,\infty}\right) \right\},$$ and $$\left\| g-s_{\lambda ,g}\right\| _{n}\leq \left\| g-s_{0,g}\right\| _{n}+\left\| s_{0,g}-s_{\lambda, g}\right\| _{n}.$$ By Lemma \[LEM:bias\], $$\left\| g-s_{0,g}\right\| _{n} \leq \left\| g-s_{0,g}\right\| _{\infty}=O_{P}\left(\frac{F_{2}}{F_{1}}\left| \triangle \right| ^{\ell +1}\left|g\right| _{\ell+1,\infty}\right).$$ Thus, the desired result is established. $\square $ Under Assumptions (A1), (A2), (C4), (C5), there exist constants $0 < c_{U} < C_{U} < \infty $, such that with probability approaching 1 as $n\rightarrow \infty$, $c_{U}\mathbf{I}_{p\times p} \leq n\mathbf{U}_{11} \leq C_{U}\mathbf{I}_{p\times p}$, where $\mathbf{U}_{11}$ is given in (\[DEF:V-inverse\]). \[LEM:U11\] *Proof.* Denote by $$\boldsymbol{\Gamma} _{\lambda }=\frac{1}{n}\left(\mathbf{B}^{\T} \mathbf{B}+\lambda\mathbf{P}\right) =\left[ \frac{1}{n}\sum_{i=1}^{n}B_{\xi }\left( \mathbf{X}_{i}\right) B_{\zeta }\left( \mathbf{X}_{i}\right) +\frac{\lambda }{n}\langle B_{\xi },B_{\zeta }\rangle _{\mathcal{E}_{\upsilon }}\right] _{\xi ,\zeta \in \mathcal{K}} \label{DEF:Gamma_lambda}$$ a symmetric positive definite matrix. Then for $\mathbf{V}_{22}$ defined in (\[DEF:V\]), we can rewrite it as $\mathbf{V}_{22}=n\mathbf{Q}_{2}^{\T}\boldsymbol{\Gamma} _{\lambda } \mathbf{Q}_{2}$. Let $\alpha _{\min}(\lambda )$ and $\alpha _{\max}(\lambda )$ be the smallest and largest eigenvalues of $\boldsymbol{\Gamma}_{\lambda }$. As shown in the proof of Theorem 2 in the Supplement of Lai and Wang (2013), there exist positive constants $0<c_3<C_3$ such that under Conditions (C4) and (C5), with probability approaching 1, we have $$c_3|\triangle |^{2} \leq \alpha _{\min}(\lambda ) \leq \alpha _{\max}(\lambda ) \leq C_3\left(|\triangle |^{2}+\frac{\lambda}{n|\triangle |^{2}}\right).$$ Therefore, we have $$c_{4}\left(|\triangle |^{2}+\frac{\lambda}{n|\triangle |^{2}}\right)^{-1} \|\mathbf{a}\|^{2} \leq n \mathbf{a}^{\T}\mathbf{V}_{22}^{-1}\mathbf{a} = \mathbf{a}^{\T}(\mathbf{Q}_2^{\T} \boldsymbol{\Gamma} _{\lambda}\mathbf{Q}_2)^{-1}\mathbf{a} \leq C_{4}|\triangle |^{-2} \|\mathbf{a}\|^{2}.$$ Thus, by Assumption (A2), we have with probability approaching 1 $$c_{5}\left(|\triangle |^{2}+\frac{\lambda}{n|\triangle |^{2}}\right)^{-1}|\triangle |^{2}\|\mathbf{a}\|^{2} \leq \mathbf{a}^{\T}\mathbf{V}_{12}\mathbf{V}_{22}^{-1} \mathbf{V}_{21}\mathbf{a} =\mathbf{a}^{\T}\mathbf{Z}^{\T}\mathbf{B}\mathbf{Q}_2\mathbf{V}_{22}^{-1} \mathbf{Q}_2^{\T}\mathbf{B}^{\T}\mathbf{Z}\mathbf{a} \leq C_{5}\|\mathbf{a}\|^{2}. \label{EQ:eigen}$$ According to (\[DEF:V-inverse\]) and (\[DEF:U11\]), we have $$(n\mathbf{U}_{11})^{-1}=n^{-1}(\mathbf{V}_{11}-\mathbf{V}_{12}\mathbf{V}_{22}^{-1}\mathbf{V}_{21}) =n^{-1}(\mathbf{Z}^{\T}\mathbf{Z} -\mathbf{V}_{12}\mathbf{V}_{22}^{-1} \mathbf{V}_{21}).$$ The desired result follows from Assumptions (A1), (A2) and (\[EQ:eigen\]). $\Box$ 0.10in **A.2. Proof of Theorem 1** Let $\mu_i=\mathbf{Z}_{i}^{\T}\boldsymbol{\beta}_{0}+g_{0}(\mathbf{X}_{i})$, $\boldsymbol{\mu}^{\T}=(\mu_1,\ldots,\mu_n)$, and let $\boldsymbol{\epsilon}^{\T}=(\epsilon_1,\ldots,\epsilon_n)$. Define $$\begin{aligned} \widetilde{\boldsymbol{\beta }}_{\mu}&=& \mathbf{U}_{11} \mathbf{Z}^{\T}\left(\mathbf{I}-\mathbf{B}\mathbf{Q}_2\mathbf{V}_{22}^{-1} \mathbf{Q}_2^{\T}\mathbf{B}^{\T}\right)\boldsymbol{\mu}, \label{betatilde_mu}\\ \widetilde{\boldsymbol{\beta }}_{\epsilon}&=& \mathbf{U}_{11} \mathbf{Z}^{\T}\left(\mathbf{I}-\mathbf{B}\mathbf{Q}_2\mathbf{V}_{22}^{-1} \mathbf{Q}_2^{\T}\mathbf{B}^{\T}\right)\boldsymbol{\epsilon}. \label{betatilde_e}\end{aligned}$$ Then $\widehat{\boldsymbol{\beta }}-\boldsymbol{\beta }_{0}=( \widetilde{\boldsymbol{\beta }}_{\mu}-\boldsymbol{\beta }_{0})+\widetilde{\boldsymbol{\beta }}_{\epsilon}$. \[LEM:betamu-beta\] Under Assumptions (A1), (A2), (C1)-(C5), $\|\widetilde{\boldsymbol{\beta }}_{\mu}-\boldsymbol{\beta }_{0}\| =o_{P}\left(n^{-1/2}\right)$ for $\widetilde{\boldsymbol{\beta }}_{\mu}$ in (\[betatilde\_mu\]). *Proof.* Let $\mathbf{g}_{0}=(g_{0}(\mathbf{X}_{1}),\ldots, g_{0}(\mathbf{X}_{n}))^{\T}$. It is clear that $$\begin{aligned} \widetilde{\boldsymbol{\beta}}_{\mu}-\boldsymbol{\beta}_{0}&=& \mathbf{U}_{11} \mathbf{Z}^{\T}\left(\mathbf{I}-\mathbf{B}\mathbf{Q}_2\mathbf{V}_{22}^{-1} \mathbf{Q}_2^{\T}\mathbf{B}^{\T}\right)\mathbf{g}_{0}\\ &=&\mathbf{U}_{11}\mathbf{Z}^{\T}\left[\mathbf{g}_{0}- \mathbf{B}\mathbf{Q}_2 \{\mathbf{Q}_2^{\T}(\mathbf{B}^{\T} \mathbf{B}+\lambda\mathbf{P})\mathbf{Q}_2\}^{-1} \mathbf{Q}_2^{\T}\mathbf{B}^{\T}\mathbf{g}_{0} \right]\\ &=&n\mathbf{U}_{11}\mathbf{A},\end{aligned}$$where $\mathbf{A}=\left( A_{1},\ldots ,A_{p}\right)^{\T}$, with $$A_{j}=n^{-1}\mathbf{Z}_{j}^{\T}\left[\mathbf{g}_{0}- \mathbf{B}\mathbf{Q}_2\{\mathbf{Q}_2^{\T}(\mathbf{B}^{\T} \mathbf{B}+\lambda\mathbf{P})\mathbf{Q}_2\}^{-1} \mathbf{Q}_2^{\T}\mathbf{B}^{\T} \mathbf{g}_{0} \right]$$for $\mathbf{Z}_{j}^{\T}=(Z_{1j},...,Z_{nj})$. Next we derive the order of $A_{j}$, $1\leq j\leq p$, as follows. For any $g_{j}\in \mathbb{S}$, by (\[EQ:pls\]) we have $$A_{j}=\left\langle z_{j}, g_{0} -s_{\lambda,g_{0}} \right\rangle_{n} =\left\langle z_{j}-g_{j},g_{0} -s_{\lambda,g_{0}} \right\rangle_{n} +\frac{\lambda}{n}\left\langle s_{\lambda,g_{0}}, g_{j} \right\rangle_{\mathcal{E}_{\upsilon}}.$$ For any $j=1,\ldots, p$, let $h_{j}(\cdot)$ be the function $h(\cdot)$ that minimizes $E\{ Z_{ij}-h(\mathbf{X}_{i})\} ^{2}$ as defined in (\[EQ:h\_j\]). According to Lemma \[LEM:appord\], there exists a function $\widetilde{h}_{j}\in \mathbb{S}$ satisfy $$\|\widetilde{h}_{j}-h_{j}\|_{\infty}\leq C \frac{F_{2}}{F_{1}}\left| \triangle \right| ^{\ell +1}\left|h_{j}\right|_{\ell+1,\infty} , \label{EQ:h_j_tilde}$$ then $$A_{j}=\langle z_{j}-h_{j}, g_{0} - s_{\lambda,g_{0}} \rangle_{n}+\langle h_{j}-\widetilde{h}_{j}, g_{0} -s_{\lambda,g_{0}} \rangle_{n} +\frac{\lambda}{n}\langle s_{\lambda,g_{0}},
{'source': 'arXiv_out/1605'}