Reproducing kernels and choices of associated feature spaces, in the form of $L^{2}$-spaces
Palle Jorgensen Feng Tian
math.FA math.PR 47L60, 46N30, 46N50, 42C15, 65R10, 05C50, 05C75, 31C20, 60J20
  (Primary), 46N20, 22E70, 31A15, 58J65, 81S25, 68T05 (Secondary)


Interpolation and reproducing kernels]Reproducing kernels and choices of associated feature spaces, in
the form of L^2-spaces



(Palle E.T. Jorgensen) Department of Mathematics, The University
of Iowa, Iowa City, IA 52242-1419, U.S.A. 

palle-jorgensen@uiowa.edu

http://www.math.uiowa.edu/~jorgen/



(Feng Tian) Department of Mathematics, Hampton University, Hampton,
VA 23668, U.S.A.

feng.tian@hamptonu.edu

Motivated by applications to the study of stochastic processes, we
introduce a new analysis of positive definite kernels K, their
reproducing kernel Hilbert spaces (RKHS), and an associated family
of feature spaces that may be chosen in the form L^2(μ);
and we study the question of which measures μ are right for a
particular kernel K. The answer to this depends on the particular
application at hand. Such applications are the focus of the separate
sections in the paper.



[2000]Primary 47L60, 46N30, 46N50, 42C15, 65R10, 05C50, 05C75, 31C20, 60J20;
Secondary 46N20, 22E70, 31A15, 58J65, 81S25, 68T05.



[
    Feng Tian
    December 30, 2023
=====================





§ INTRODUCTION


The use of reproducing kernels and their reproducing kernel Hilbert
spaces (RKHSs) was initially motivated by problems in classical analysis,
and it was put into an especially attractive and useful form by Aronszajn
in the 1950ties. Since then the applications of kernel theory has
greatly expanded, both in pure and applied mathematics. An application
of more recent vintage is machine learning. The number of applied
areas include use of RKHSs in the study of stochastic processes, especially
as a tool in Ito calculus; and in machine learning (ML). The last
two are related, and they are the focus of our present paper. Dictated
by a number of practical applications of the theory of ML, starting
with a positive definite (p.d.) kernel K, it has proved useful
to study both the associated RKHS itself, as well as a variety of
choices of feature spaces (for details, see p1 inside the
paper); and the interplay between them.

Now motivated by related applications to the study of stochastic processes,
it is of special significance to focus on the cases when the family
of feature spaces may be chosen in the form L^2(μ);
but this then raises the question of which measures μ are right
for a particular kernel K, and its associated RKHS. The answer
to this depends on the particular application at hand. Such applications
are the focus of the separate sections below inside the paper. 

In our study of RKHSs and choices of feature spaces, we have focused
on those of especial relevance to analysis of Gaussian calculus, but
there are many others, for example, functional and harmonic analysis,
boundary value problems, PDE, geometry and geometric analysis, operator
algebras/theory, the theory of unitary representations, mathematical
physics, and the study of fractals and fractal measures. Even this
list is not exhaustive. Nonetheless, we have narrowed our scope, and
our choice of applications, for the present paper. The reader will
be able to follow up on the various other directions, not covered
here, with the use of our cited references, see especially our discussion
of the literature below.

Discussion of the literature. The theory of RKHS and their
applications is vast, and below we only make a selection. Readers
will be able to find more cited there. As for the general theory of
RKHS in the pointwise category, we find useful <cit.>.
The applications include fractals (see e.g., <cit.>);
probability theory <cit.>;
and learning theory <cit.>.



§ REPRODUCING KERNELS


The present setting begins with a fixed positive definite (p.d.) kernel
K, i.e., a function K:S× S⟶ℝ where
S is a set, and satisfying

    ∑ _i∑ _jα_iα_jK(s_i,s_j)≥0

for all {α_i} _1^n, { s_i} _1^n,
α_i∈ℝ, s_i∈ S, and n∈ℕ. 

Even though we shall state our definitions and results in the special
case of real valued functions, the complex case will result from our
present setting with only minor modifications. But in order to minimize
technical points, we have restricted the present discussion to the
real case. 

The two more general settings are as follows: (i) complex; and (ii)
operator valued.


  * There the definition is as in (<ref>), but now K:S× S⟶ℂ,
and the p.d. assumption is instead:

    ∑ _i∑ _jα_iα_jK(s_i,s_j)≥0

for all choices of {α_i} and { s_i},
α_i∈ℂ, s_i∈ S, 1≤ i≤ n. 

  * Let H be a complex Hilbert space, and let ℬ(H)=
the algebra of all bounded linear operators in H, i.e., H⟶ H.
In this case, our setting for the kernel K is: K:S× S⟶ℬ(H),
and now we assume instead that, for all s_i, h_i, with s_i∈ S,
h_i∈ H, and 1≤ i≤ n; and all n∈ℕ, we
have: 

    ∑ _i∑ _j⟨ K(s_i,s_j)h_i,h_j⟩ _H≥0.





Given a positive definite (p.d.) kernel K on S,
we shall consider pairs (F,𝐇) where 𝐇
is a Hilbert space, and F:S⟶𝐇 is a function
satisfying 

    ⟨ F(s),F(t)⟩ _𝐇=K(s,t), ∀ s,t∈ S.

If (F,𝐇) satisfies this, we say that 𝐇
is a feature space, or a feature Hilbert space. 



In a general setup, reproducing kernel Hilbert spaces (RKHSs) were
pioneered by Aronszajn in the 1950s <cit.>; and
subsequently they have been used in a host of applications. The key
idea of Aronszajn is that a RKHS is a Hilbert space ℋ(K)
of functions f on a set such that the values f(x) are “reproduced”
from f and a vector K_x in ℋ(K), in
such a way that the inner product ⟨ K_x,K_y⟩=:K(x,y)
is a positive definite kernel. 

By a theorem of Kolmogorov, every Hilbert space may be realized as
a (Gaussian) reproducing kernel Hilbert space (RKHS), see e.g., <cit.>,
and the details below.


Let (Ω,𝒞,ℙ) be a probability
space. We will be interested in centered Gaussian processes (X_s)_s∈ S
(see e.g., <cit.>), indexed by S, satisfying


  * X_s is Gaussian w.r.t. a probability space (Ω,𝒞,ℙ),

  * X_s∈ L^2(Ω,ℙ), and

    𝔼(X_s)     =0,
    𝔼(X_sX_t)     =K(s,t), ∀ s,t∈ S,

where 𝔼 denotes the expectation with respect to ℙ. 

Given a p.d. kernel K, it is well known that a Gaussian realization
as in (<ref>)-(<ref>) always exists; in fact, we may
choose ℙ such that Ω=ℝ^S= all functions
on S, 𝒞= the corresponding cylinder σ-algebra
of subsets of Ω; and 

    X_s(ω):=ω(s), ∀ω∈Ω, s∈ S.


A p.d. kernel of particular interest in the present paper will be
as follows: 

Let (V,ℬ,μ) be a measure space, where μ
is assumed positive and σ-finite. Set ℬ_fin:={ A∈ℬ;μ<∞}.
On ℬ_fin×ℬ_fin, then define K^(μ)
by 

    K^(μ)(A,B):=μ(A∩ B), ∀ A,B∈ℬ_fin.


It is immediate that K^(μ) is p.d., and there is
therefore a canonical associated centered Gaussian process X=X^(μ),
indexed by ℬ_fin, satisfying 

    𝔼(X_A^(μ)X_B^(μ))=μ(A∩ B), ∀ A,B∈ℬ_fin.

We shall study this process in detail and show that it may be used
to interpolate any Markov process built on (V,ℬ);
see m1. 

A tool in our analysis will be reproducing kernel Hilbert spaces (RKHSs).
Recall that every p.d. kernel K has an associated and unique RKHS
ℋ(K). The reproducing axiom is as follows:
K(·,s)∈ℋ(K), and 

    F(s)=⟨ F,K(·,s)⟩ _ℋ(K), ∀ s∈ S, for .


We now present two general lemmas, applied to any p.d. kernel K:S× S⟶ℂ.
Let ℋ(K) be the corresponding RKHS. 

A function ψ on S is in ℋ(K)
⟺ ∃ C=C_ψ<∞, a finite constant,
s.t. for ∀ n∈ℕ, ∀{ s_i} _1^n,
s_i∈ S, ∀{α_i} _1^n, α_i∈ℂ,
we have 

    |∑ _iα_iψ(s_i)|^2≤ C_ψ∑ _i∑ _jα_iα_jK(s_i,s_j).




This is standard Aronszajn theory <cit.>.
If (<ref>) holds, then we may take the constant C_ψ=‖ψ‖ _ℋ^2,
and it is the smallest choice of admissible constant. 


 Let K_1 and K_2: S× S⟶ℂ
both be p.d.; and let ℋ(K_i), i=1,2,
be the corresponding RKHSs. Then the following are equivalent:


  * ℋ(K_1)⊆ℋ(K_2);

  * ∃ C<∞ such that for ∀ n∈ℕ, ∀{ s_i} _1^n,
{α_i} _1^n, s_i∈ S, α_i∈ℂ,
we have the estimate:

    ∑ _i∑ _jα_iα_jK_1(s_i,s_j)≤ C∑ _i∑ _jα_iα_jK_2(s_i,s_j).

Stated equivalently, CK_2-K_1 is positive definite (p.d.). 




This follows immediately from (<ref>) and rk1.





§ APPLICATION TO WHITE NOISE ANALYSIS


White noise analysis serves as a versatile framework for stochastic
and infinite-dimensional analysis, with a growing number of applications
to neighboring areas, probability, mathematical statistics, and quantum
physics. The setting is that of (Gaussian, continuous parameter) white
noise — a generalized random process indexed by elements
in a σ-algebra and with independent values at disjoint sets;
informally, we may view it as an infinite system of coordinates on
which to base an infinite-dimensional calculus. More precisely, the
starting point is the L^2-space of a white noise measure (Wiener
measure). A common approach makes use of a certain choice of a Gelfand
triples <cit.>. Our approach is both entirely different,
and it is more general. The wider aim is an infinite-dimensional differential
calculus, and calculus of variation. 

Let (V,ℬ) be a measure space, and let μ
be a σ-finite measure on (V,ℬ); then
define K=K^(μ) as follows: K^(μ):ℬ_fin×ℬ_fin⟶ℝ,

    K^(μ)(A,B)=μ(A∩ B), ∀ A,B∈ℬ_fin,

where ℬ_fin={ A∈ℬ;μ(A)<∞}. 

K^(μ) as in (<ref>) is positive definite. 



We have 

    ∑ _i∑ _jα_iα_jK^(μ)(A_i,A_j)     =‖∑ _iα_iK^(μ)(·,A_j)‖ _ℋ(K^(μ))^2
        =∫|∑ _iα_i1_A_i|^2dμ≥0,

for ∀{α_i} _1^n, α_i∈ℝ,
∀{ A_i} _1^n, A_i∈ℬ_fin,
∀ n∈ℕ. 


Let (V,ℬ,μ) be a measure
space, μ assumed σ-finite (positive). Let K^(μ)(A,B):=μ(A∩ B),
A,B∈ℬ_fin, be the corresponding p.d. kernel; and
let ℋ(K^(μ)) be the RKHS. Then


    ℋ(K^(μ))     ={ F signed measures on (V,ℬ) s.t. 
            dF≪ dμ(abs. cont) with dF/dμ∈ L^2(μ)}; and
    ‖ F‖ _ℋ(K^(μ))    =‖dF/dμ‖ _L^2(μ).





We may use rk1 to show that F, as in (<ref>),
is indeed in ℋ(K^(μ)). 

Assume F is as specified in (<ref>); and set φ=dF/dμ
(∈ L^2(μ)), which is the condition from (<ref>)
on the Radon-Nikodym derivative. We will show that, if n∈ℕ,
(A_i)_1^n, A_i∈ℬ_fin (i.e.,
μ(A_i)<∞), α_i∈ℝ, then

    |∑ _iα_iF(A_i)|^2≤‖φ‖ _L^2(μ)^2∑ _i∑ _jα_iα_j=μ(A_i∩ A_j)K^(μ)(A_i,A_j)

and so we conclude that F∈ℋ(K^(μ)),
with ‖ F‖ _ℋ(K^(μ))≤‖φ‖ _L^2(μ).
It is in fact “=”. See below. 

We now give the verification of (<ref>): Let n, (A_i)_1^n,
(α_i)_1^n, and φ:=dF/dμ
∈ L^2(μ) be as stated in (<ref>), and
the discussion above; then 

    LHS_(<ref>)    =    |∑ _iα_iF(A_i)|^2=|∑ _iα_i∫_A_iφ dμ|^2
        =    |∫_Vφ·∑ _iα_i1_A_idμ|^2
       (Schwarz)≤   ‖φ‖ _L^2(μ)^2‖∑ _iα_i1_A_i‖ _L^2(μ)^2
        =    ‖φ‖ _L^2(μ)^2∑ _i∑ _jα_iα_jμ(A_i∩ A_j)

which is the desired conclusion (<ref>).

Every F∈ℋ(K^(μ)) is a σ-additive
signed measure, i.e., if A=∪_i=1^∞A_i, A_i∩ A_j=∅,
i≠ j; (sets in ℬ_fin) then 

    F(A)=∑ _i=1^∞F(A_i).




Proof of (<ref>). 

LHS_(<ref>)     =F(A)=⟨F,μ(·∩A)⟩_ℋ(K^(μ)),           μ(·∩A)=K^(μ)(·,A),

     =⟨F,∑_i=1^∞μ(·∩A_i)⟩_ℋ(K^(μ)),          μ(·∩A_i)=K^(μ)(·,A_i),

     =∑_i=1^∞⟨F,K^(μ)(·,A_i)⟩_ℋ(K^(μ))

     =∑_i=1^∞F(A_i).

We used K^(μ)(·,A_i)⊥ K^(μ)(·,A_j)
for i≠ j. 



For A,B∈ℬ_fin, we have 

    dK^(μ)(·,A)/dμ=1_A=the indicator function.




Proof. For A,B∈ℬ_fin, we have 

    K^(μ)(A,B)=∫_B1_A(x)dμ(x)=μ(B∩ A),

and (<ref>) follows.



If F∈ℋ(K^(μ)), then dF≪ dμ where
dF is the signed measure in (<ref>).



Proof. We show that [μ(A)=0] ⟹
[F(A)=0]. From the reproducing property in
ℋ(K^(μ)), we have:

    F(A)=⟨ F,K^(μ)(·,A)⟩ _ℋ(K^(μ))=⟨ F,μ(·∩ A)⟩ _ℋ(K^(μ));

hence, μ(A)=0 ⟹ F(A)=0,
since μ(A)=0 ⟹ K^(μ)(·,A)=0,
and so F(A)=⟨ F,K^(μ)(·,A)⟩ _ℋ(K^(μ))=0. 



The proof of n1 is complete. 



Let (V,ℬ,μ) be a fixed σ-finite
measure space, and let ℋ(K^(μ)) be the
RKHS of K^(μ)(A,B):=μ(A∩ B),
A,B∈ℬ_fin (see (<ref>)). Define W^(μ)
as an isometry: W^(μ):L^2(μ)∋ f⟼ fdμ∈ℋ(K^(μ)),
where W^(μ)(f)=fdμ is a signed measure
on (V,ℬ); then 

    W^(μ):L^2(μ)≃ℋ(K^(μ))

is an isometric isomorphism onto ℋ(K^(μ)). 



Let (V,ℬ) be a measure space,
μ a σ-finite measure on ℬ , and set ℬ_fin={ A∈ℬ;μ(A)<∞}.
Let K be a p.d. kernel on ℬ_fin×ℬ_fin,
and let ℋ(K) be the corresponding RKHS. Suppose
ℋ(K) consists of signed measures; and set

    ℋ(μ):={ F∈ℋ(K);dF≪ dμ, and dF/dμ∈ L^2(μ)} .

Then 

    ℋ(μ)⊆ℋ(K^(μ)),

where 

    K^(μ)(A,B):=μ(A∩ B), ∀ A,B∈ℬ_fin;

and therefore ∃ c(μ)<∞ such that 

    c(μ)K^(μ)-K

is positive definite. 



Let F∈ℋ(μ), see (<ref>); and
set φ^(F):=dF/dμ. Let n∈ℕ,
{ A_i} _1^n, A_i∈ℬ_fin, {α_i} _1^n,
α_i∈ℝ; then 

    |∑ _iα_iF(A_i)|^2    =    |∑ _iα_i∫_A_iφ^(F)(x)dμ(x)|^2
        =    |∫_V(∑ _iα_i1_A_i)φ^(F)dμ|^2
       (Schwarz)≤   ∑ _i∑ _iα_iα_jμ(A_i∩ A_j)‖φ^(F)‖_L^2(μ)^2.

Hence by rk1, F∈ℋ(K^(μ)),
see (<ref>); and ‖ F‖ _ℋ(K^(μ))≤‖φ_L^2(μ)^(F)‖. 

Conclusion (<ref>) now follows. Finally conclusion (<ref>)
is immediate from rk2.


If V=[0,∞), ℬ= Borel σ-algebra,
μ=dx=λ, Lebesgue measure, then X^(μ)=
standard Brownian motion. 



Let A=[0,t], B=[0,s], s,t∈[0,∞),
then X_A^(μ)=W_[0,t], X_B^(μ)=W_[0,s]
satisfying 

    E(X_A^(μ)X_B^(μ))=𝔼(W_[0,t]W_[0,s])=λ([0,t]∩[0,s])=s∧ t,

so the standard p.d. kernel which determines Brownian motion; and
d[W_0,t]_2=dt, and [W_0,t]_2=t,
referring to the quadratic variation, see also qa and qv. 

Recall that in general, if K is a p.d. kernel, ℋ(K)
the RKHS, then whenever F:S⟶ℋ is a function
from S into a Hilbert space ℋ s.t. K(t,s)=⟨ F(s),F(t)⟩ _ℋ,
there is then a corresponding transform L=L_F:ℋ⟶ℋ(K),
given by 

    (Lh)(t)=⟨ h,F(t)⟩ _ℋ, ∀ t∈ S, ∀ h∈ℋ.

In bm1, we may apply this to this to the kernel K=K^(μ),
S=ℬ_fin, K^(μ)(A,B):=μ(A∩ B),
A,B∈ℬ_fin, and let F(A):=X_A^(μ),
A∈ℬ_fin. 

 Let (V,ℬ,μ) be as in bm1.
Let Ω:=ℝ^ℬ_fin, and set F:ℬ_fin⟶ L^2(Ω,𝒞,ℙ),

    F(A):=X_A^(μ), A∈ℬ_fin,

where X_A^(μ)∈ L^2(Ω,ℙ)
is the centered Gaussian process with covariance kernel 𝔼(X_A^(μ)X_B^(μ))=μ(A∩ B). 

Then the transform L:L^2(Ω,ℙ)⟶ℋ(K^(μ))
is 

    (Lh)(A)=𝔼(hX_A^(μ)), ∀ A∈ℬ_fin, ∀ h∈ L^2(Ω,ℙ).


Let ℱ_V:= all measurable functions in (V,ℬ),
and f∈ L^2(μ)⊂ℱ_V (real valued),
we get the Ito-integral 

    ∫_VfdX:=lim∑_if(s_i)X_A_i^(μ),

where the limit is taken over all measurable partitions of V, mesh
→0. Then 

    𝔼(|∫_VfdX|^2)=∫|f|^2dμ.




(sketch) For all partitions { A_i} on V, s_i∈ A_i,
the Ito-isometry (<ref>) follows from the approximation:

    𝔼(|∑ _if(s_i)X_A_i^(μ)|^2)=∑ _i|f(s_i)|^2μ(A_i)∫_V|f|^2dμ.



Using this version of Ito integral, we get the following conclusions. 

Given a fixed σ-finite measure space (V,ℬ,μ),
set K=K^(μ), 

    K^(μ)(A,B)=μ(A∩ B), A,B∈ℬ_fin,

and let L^2(Ω,𝒞,ℙ) be the
corresponding probability space s.t. 

    X_A^(μ)(ω)=ω(A), ω∈Ω=ℝ^ℬ.

Then 

    𝔼(X_A^(μ)X_B^(μ))=K^(μ)(A,B)=μ(A∩ B),

and the Ito integral X_f^(μ)=∫_VfdX is well
defined with 

    𝔼(|X_f^(μ)|^2)=∫_V|f|^2dμ,

and 

    μ=QV(X)=[X,X]=[X]_2;

see also qa and qv. 

The correspondence {X_A^(μ)}_A∈ℬ⟷{X_f^(μ)}_f∈ L^2(μ)
is bijective. Easy direction: given X_f^(μ) as
above, A∈ℬ, set f=1_A. 

For details on Ito calculus and Brownian motion, see, e.g., <cit.>. 



Given (V,ℬ,μ) fixed, σ-finite measure
space, we introduce the kernel K^(μ), and the associated
centered Gaussian process X:=X^(μ). From our Ito-calculus,
it follows that X^(μ) may be realized in two equivalent
ways: 


  * Ω=ℝ^ℬ= all functions
from ℬ into ℝ, X_A^(μ)(ω)=ω(A),
A∈ℬ;

  * Ω=ℝ^V= all functions from V
into ℝ, X_f^(μ)(ω)=ω(f),
f∈ L^2(μ). 

From standard Kolmogorov consistency theory <cit.>,
in (<ref>) the probability measure ℙ is defined
on the cylinder σ-algebra of ℝ^ℬ,
and in case (<ref>) it is defined on the σ-algebra
for ℝ^V. 

We also get two equivalent versions of the covariance function for
X, which is indexed by ℬ or by L^2(μ): 


  * 

𝔼(X_AX_B)=μ(A∩ B),
A,B∈ℬ, 






    ⇕




  * 𝔼(X_fX_g)=∫_Vf(x)g(x)dμ(x)=𝔼((∫ fdX)(∫ gdX)),
∀ f,g∈ L^2(μ), real valued, where X_f=∫ fdX
is the Ito integral formula which made the link from (<ref>)
to (<ref>).




Let X:=X^(μ) be the Gaussian process as above, and

    X_f:=∫_VfdX,

then 

    𝔼(e^iX_f)=e^-1/2∫_V|f|^2dμ, ∀ f∈ L^2(μ).




Direct proof from the power series expansions. See, e.g., <cit.>,
and the papers cited there.


Note that (<ref>) is analogous to the Gelfand triple construction,
but more general. In the present setting, we do not need a Gelfand
triple in order to make process (<ref>) above. 



If { f_n} _n∈ℕ_0 is an orthonormal
basis (ONB) in L^2(μ), then { X_f_n} _n∈ℕ_0
is an i.i.d. N(0,1) system, i.e., X_f_n=Z_n∼ N(0,1),
and the following Karhunen-Loeve decomposition holds: 

    X_A=∑_n=0^∞(∫_Af_ndμ)Z_n, ∀ A∈ℬ_fin.




Assume μ is non-atomic. Then the quadratic variation
of X:=X^(μ) is μ itself, i.e., if B∈ℬ,
d[X,X]=dμ. 



Let B∈ℬ with a partition { A_i}
s.t. B=∪ A_i, A_i∩ A_j=∅, i≠ j. If
μ is non-atomic, then 

    =:[X,X]=[X]_2lim∑_i(X_A_i)^2=μ(B)1

where 1 denotes the constant function in L^2(Ω,ℙ),
and the limit is over the set of all partitions of B with mesh
tending to 0. See qv for additional details.


Let f:ℝ⟶ℝ, or ℂ, f∈ C^2,
then 

    df(X_s)=f'(X_s)dX_s+1/2f”(X_s)dμ(s),

or equivalently, 

    f(X_B)=∫_Bf'(X_s)dX_s+1/2∫_Bf”(X_s)dμ(s),

for ∀ B∈ℬ_fin, where we used the Ito integral,
and d[X,X](s)=dμ(s) for the
quadratic variation. 



We can do most of the white noise analysis in the more general setting,
i.e., w.r.t (<ref>). 



Let the setting be as in n2, i.e., (V,ℬ,μ),
and K are specified as in (<ref>)-(<ref>). In
particular, in addition to K, we also have the μ-kernel K^(μ)(A,B)=μ(A∩ B)
as in (<ref>). Let X^(K) be the centered
Gaussian process with kernel K, i.e., 

    𝔼(X_A^(K)X_B^(K))=K(A,B), ∀ A,B∈ℬ_fin.

Let X^(μ) be the Gaussian process (<ref>)
with Ito integral 

    X_f^(μ)=∫_Vf(x)dX_x^(μ), f∈ L^2(μ),

and 

    𝔼(|X_f^(μ)|^2)=∫_V|f|^2dμ;

see et. 

Then there is a function 

    G:ℬ_fin× V⟶ℝ,

measurable in the second variable, such that

    G(A,·)    ∈ L^2(μ), ∀ A∈ℬ_fin;
    
    K(A,B)     =∫_VG(A,x)G(B,x)dμ(x)

(compare with p1 below), and 

    X_A^(K)=∫_VG(A,x)dX_x^(μ).




The existence of G follows from n2, and the Hida-Cramer
transform <cit.>. Hence, by (<ref>), we may define
a Gaussian process X^(K) by (<ref>); and
for A,B∈ℬ_fin, we have

    𝔼(X_A^(K)X_B^(K))     =     K(A,B)
       (by (<ref>))=   ∫_VG(A,x)G(B,x)dμ(x),

which is the desired conclusion.




§ GAUSSIAN INTERPOLATION OF MARKOV PROCESSES


Markov models, or hidden Markov models, are ubiquitous in model building,
e.g., to models for speech and handwriting recognition, to software,
and learning mechanisms in biological neural networks. Within the
study of support vector machines, one use of Markov processes is to
solve both the problem of classification, and that of clustering.
The list of optimization tasks includes that of maximizing an “expected
goodness of classification,” or a “goodness of clustering” criterion.
This in turn leads to the study of specific kinds of probability distribution
over sequences of vectors — for which we have good parameter
estimation, and good marginal distribution algorithms.

Hidden Markov models tend to be robust in many uses, for example,
in determining the nature of an input signal, given the corresponding
an output. The model aims to determine the most probable set of parameters
which dictate input states, when based on an observed sequence of
output states.

The literature is quite large: Here we mention just <cit.>,
and the papers cited there.



 §.§ The Markov processes


In our previous work <cit.>, we already discussed applications
of the family of Gaussian processes from wn. Our present
aim is to use them in an interpolation algorithm for non-atomic Markov
processes. 

Recall the Gaussian processes {X_A^(μ);A∈ℬ_fin},
such that 

    𝔼(X_A^(μ)X_B^(μ))=μ(A∩ B), ∀ A,B∈ℬ_fin;

where (V,ℬ,μ) is a given measure space,
and μ assumed positive and σ-finite. 

    K^(μ)[rd]
    μ@ >[rr][ur]        X^(μ)


Below we consider a family of Gaussian processes corresponding to
a given Markov process P(x,A), where (V,ℬ)
is a measure space, x∈ V, P(x,·) is a non-atomic
probability measure, i.e., P(x,V)=1. We shall denote
P as the transition operator, defined for measurable functions
f on (V,ℬ), by

    (Pf)(x)=∫_Vf(y)P(x,dy), ∀ x∈ V.

Thus P(1)=1, and the constant function
1 is harmonic. (Also see <cit.>, and
the papers cited there.)

Every generalized Markov process P(x,·) induces
a dual pairs of actions: 


  * action on measurable functions f on (V,ℬ),

    f⟼∫ f(y)P(x,dy)=(Pf)(x), x∈ V; and


  * action on signed measures ν on (V,ℬ),

    ν⟼∫ P(x,·)dν(x)=P^*(ν),

where 

    (P^*(ν))(A)=∫ P(x,A)dν(x), ∀ A∈ℬ.




As in standard Markov theory, 

    P_2(x,A)=∫ P(x,dy)P(y,A)=P[P(·,A)](x),

and inductively

    P_n+1(x,A)=∫ P_n(x,dy)P(y,A).


For each of the measures P(x,·),P_2(x,·)⋯,P_n(x,·),
there is a corresponding white noise process X^(x),
i.e., an indexed family of Gaussian processes X_A^(x)
∼ P(x,A), where 𝔼_x(X_A^(x))=0,
and 

    𝔼_x(X_A^(x)X_B^(x))=P(x,A∩ B), ∀ A,B∈ℬ.

We now introduce a more general family of Ito integrals, and get a
new process W_A^(x) which has P_2(x,A)
as its covariance kernel. See m1 below. 

 Let P(x,·) and X_A^(x)
be as specified above, see (<ref>)-(<ref>). Set 

    W_A^(x):=∫_VX_A^(y)dX^(x)(y),

defined as an Ito integral. Then


  * {W_A^(x)} is a Gaussian process;

  * 𝔼_x(W_A^(x))=0;

  * 𝔼_x(W_A^(x)W_B^(x))=P_2(x,A∩ B),
∀ A,B∈ℬ, ∀ x∈ V. 

  * By induction, with an n-fold Ito integral from (<ref>), we
get W_A^(n)(x) such that 

    𝔼_x(W_A^(n)(x)W_B^(n)(x))=P_n(x,A∩ B),

for n∈ℕ, x∈ V, and A,B∈ℬ. 




Let P(x,A) and X_A^(x) be as specified
above. We then form the Ito integral X_(·)^(x)
with P(x,·) as covariance. Note that for every y∈ V,
X^(y) is a centered Gaussian process with covariance
kernel 

    𝔼_y(X_A^(y)X_B^(y))=P(y,A∩ B).

We shall show that W_(·)^(x) is also
a centered (i.e., mean zero) Gaussian process, now with P_2(x,·)
as covariance measure, i.e., that W^(x) from (<ref>)
will satisfy:

    𝔼_x(W_A^(x)W_B^(x))=P_2(x,A∩ B), ∀ A,B∈ℬ;


The idea is that the white noise process interpolates the Markov process.
Aside from the induction, the key step in the argument is an analysis
of the Ito integral (<ref>). By general Ito theory, we have

    𝔼(|W_A^(x)|^2)     =∫𝔼(|W_A^(y)|^2)d[X^(x),X^(x)](y)
        =∫ P(y,A)P(x,dy)=P_2(x,A).

In the last step, we used (<ref>) on the term 𝔼(|X_A^(y)|^2)
in (<ref>); and used the formula for the quadratic variation

    d[X^(x),X^(x)](y)=P(x,dy).

See also qv below.

Note that (<ref>) is a special case of an analogous property
of white noise, subject to a fixed measure μ. Assume μ is
non-atomic, then 

    d[X^(μ),X^(μ)](y)=dμ(y),

for x fixed. We apply this to dμ(y)=P(x,dy),
and X^(x)∼ P(x,·).

Give (V,ℬ,μ) as in above,
and let X:=X^(μ) be the corresponding Gaussian process,
centered, with covariance given by (<ref>). Let d[X,X]
be the quadratic variation measure, i.e., for B∈ℬ,
QV(B)=lim∑ X_A_i^2, where the limit is taken
over all measurable partitions π={ A_i} of B,
as mesh(π)→0. Then μ(B)=QV(B),
and dμ=d[X,X]. 



It is true in general that if (V,ℬ,μ) is
a σ-finite non-atomic measure space, and X=X^(μ)
is the white noise Gaussian process determined by 

    𝔼(X_AX_B)=μ(A∩ B), ∀ A,B∈ℬ;

then for the quadratic variation measure [X,X]=[X]_2
, we have:

    [X]_2(B)=μ(B),

and so 

    d[X]_2(s)=dμ(s).


To see this, fix B∈ℬ, and take a limit on all measurable
partitions π={ A_i}, where A_i∩ A_j,
i≠ j, ∪ A_i=B, and set α_i=μ(A_i).
A direct calculation gives 

    𝔼(X_A_i^4)=3(𝔼(X_A_i^2))^2=3α_i^2, and


    𝔼(|μ(B)-∑ _iX_A_i^2|^2)
        =    𝔼(|∑ _iμ(A_i)-∑ _iX_A_i^2|^2)
        =    ∑ _i𝔼(|μ(A_i)-X_A_i^2|^2)+∑ _i≠ j𝔼((μ(A_i)-X_A_i^2)(μ(A_j)-X_A_j^2))
        =     2∑ _iμ(A_i)^2=2∑ _iα_i^20,

since ∑_iα_i=μ(B)>0 is fixed. 

In the proof of qv, we used the following fact:

Let n∈ℕ be fixed, and let α_i>0 satisfying
∑_iα_i=1, then 

    inf_n{∑ _1^nα_i^2;∑ _1^nα_i=1} =0.




Fix n∈ℕ, and apply Lagrange multiplier to get α_i=1/n,
for all i. Then 

    ∑_1^nα_i^2=1/n0.





 §.§ Fourier duality


Let (V,ℬ,μ)=(ℝ,ℬ,μ)
where μ is a probability measure on ℝ. Let X=X^(μ)
be the corresponding Gaussian process, and use the Ito integral to
define 

    X_e_t=∫_ℝe_t(x)dX_x, where   e_t(x)=exp(i2π xt), x,t∈ℝ.

Then 

    𝔼(X_e_tX_e_s)=μ̂(t-s), t,s∈ℝ,

where μ̂ denotes the standard Fourier transform. 



Direct computation using (<ref>): 

    𝔼(X_e_tX_e_s)=∫ e_t(x)e_s(x)d[X]_2=∫ e_t-s(x)dμ(x)

where we used that d[X]_2=dμ, see qv.
Then 

    ∫_ℝ∫_ℝe_t(x)e_s(x) use orthogonality𝔼(dX_x^(μ)dX_y^(μ))=∫_ℝe_t(x)e_s(x)dμ(x).

Note that if A∩ B=∅, then 𝔼(X_B^(μ)X_B^(μ))=μ(A∩ B)=μ(∅)=0. 


The Fourier transform X_e_t^(μ)=∫ e_t(x)dX_x^(μ)
is well defined, and it is a stationary process with covariance kernel

    K^F(s,t)=μ̂(t-s),

where μ̂ is the standard Fourier transform of the measure
μ.




Application. In one of our earlier papers <cit.>,
and papers cited there, we studied tempered measures μ on ℝ,
and processes { Y_φ} indexed by φ∈𝒮
(= the Schwartz space), and we get 

    𝔼(|Y_φ|^2)=∫_ℝ|φ̂(x)|^2dμ(x),

or equivalently, 

    𝔼(e^iY_φ)=e^-1/2∫|φ̂|^2dμ.

But we can recover this setting from the case (ℝ,ℬ,μ)
by setting 

    Y_φ=∫φ̂(x)dX_x^(μ)

as an Ito integral. (Note that the RHS in (<ref>) is a continuous
positive definite function in φ∈𝒮 (the Schwartz
space), and so Minlos' theorem applies; see <cit.>.) Then

    Y_φ    =∫φ̂(x)dX_x^(μ)=∬φ(t)e_t(x)dt dX_x^(μ)
        =∫φ(t)(∫e_t(x)dX_x^(μ))dt,

and 

    𝔼(|Y_φ|^2)     =𝔼(|∫φ̂(x)dX_x^(μ)|^2)
        =𝔼(|∫φ(t)dX_e_t^(μ)dt|^2)
        =∬φ(t)φ(s)𝔼(X_e_tX_e_s)dtds
        =∬φ(t)μ̂(t-s)φ(s)dtds  by &
        =∫_ℝ|φ̂(x)|^2dμ(x).

This is the desired conclusion (<ref>). The idea is that
we get all these conclusions without Gelfand triples. 



The converse holds too. Suppose {Y_φ}_φ∈𝒮
is a Gaussian process (based on ℝ) computed from a tempered
measure μ, 

    ∫_ℝdμ(x)/1+x^2<∞,

with the Gelfand triple 

    𝒮↪ L^2(ℝ)↪𝒮'

where 𝒮 is the Schwartz space, and 𝒮' the
dual of all tempered distributions. Then {Y_φ}_φ∈𝒮
is the transform of the process X^(μ) ⟺

    𝔼(|Y_φ|^2)=∫_ℝ|φ̂(x)|^2dμ(x).




Here the process X^(μ) is determined by measure
μ, 

    𝔼(X_A^(μ)X_B^(μ))     =μ(A∩ B),  ∀ A,B∈ℬ,
    
    X_f^(μ)    =∫ fdX^(μ)  ∀ f∈ L^2(ℝ,μ), and


    Y_φ=∫φ̂(x)dX_x^(μ)  ∀φ∈𝒮.

Indeed, we already proved that (<ref>) ⟹
(<ref>). Note that 

    𝔼(|Y_φ|^2)    (Ito)=∫|φ̂(x)|^2dμ(x)
        =∭φ(x)φ(t)e_x(s-t)dμ(x)dsdt
        =∬φ(s)φ(t)μ̂(s-t)dsdt.

Now set X_e_t=∫ e_t(x)dX_x^(μ),
then 

    𝔼(X_e_tX_e_s)     =μ̂(s-t), and 
    ∫φ̂(x)dX_x^(μ)    =∫φ(t)X_e_tdt, ∀φ∈𝒮.






 §.§ A stochastic bilinear form


Let (V,ℬ,μ) be a fixed σ-finite
measure as above, and let (Ω,𝒞,ℙ)
be the measure space which realizes the process {X_A^(μ)}_A∈ℬ,
and set X_f^(μ)=∫ f(x)dX_x^(μ)
(Ito integral), f∈ L^2(μ). Hence 

    𝔼(|X_f^(μ)|^2)=∫_Ω|X_f^(μ)|^2dℙ=∫_V|f|^2dμ

holds by the generalized Ito isometry, and we define the transform
L:H⟶ℋ(K^(μ)),

    (Lh)(A)=𝔼(hX_A^(μ)), ∀ A∈ℬ,

where H=L^2(Ω,ℙ), and ℋ(K^(μ))=
the RKHS of K^(μ)(A,B):=μ(A∩ B).
But using (<ref>) again, we get 

    𝔼(hX_A^(μ))=∫_Vh1_Adμ=∫_Ahdμ

for h∈ L^2(Ω,ℙ), and where 1_A,
∀ A∈ℬ, is the indicator function on A, i.e.,

    1_A(x)=δ_x(A)=
    1    if 
    
    0    else.


We have realized ℋ(K^(μ)) as
a Hilbert space of functions on (V,ℬ) viz
≃ L^2(μ). Note that since K^(μ)(A,B)=μ(A∩ B)
is a p.d. kernel on ℬ_fin×ℬ_fin, initially
ℋ(K^(μ)) is a Hilbert space
of functions on ℬ_fin, but not functions on (V,ℬ). 

Let (V,ℬ,μ) satisfy the axioms from above,
let K^(μ) be the p.d. kernel on ℬ_fin×ℬ_fin,
and ℋ(K^(μ)) be the corresponding
RKHS. Let X_f^(μ), f∈ L^2(μ),
be the Gaussian process which extends {X_A^(μ)}_A∈ℬ_fin,
and let H=L^2(Ω,ℙ) be the Gaussian Hilbert
space with inner product ⟨ h_1,h_2⟩ _H=𝔼(h_1h_2),
∀ h_i∈ H. Then the bilinear mapping 

    L^2(μ)× H∋(f,h)⟼𝔼(X_f^(μ)h)

defines two operators in duality: 

    L^2(μ)@/^1.3pc/[rr]^I_μ       H@/^1.3pc/[ll]^ξ^(μ)

The Ito isometry I_μ:L^2(μ)⟶ H,

    I_μ(f)=X_f^(μ), ∀ f∈ L^2(μ),

and the co-isometry ξ^(μ):H⟶ L^2(μ),
determined by 

    ⟨ I_μ(f),h⟩ _H=⟨ f,ξ^(μ)(h)⟩_L^2(μ), ∀ f∈ L^2(μ), ∀ h∈ H.

In particular, I_μ^*=ξ^(μ), (ξ^(μ))^*=I_μ. 


Duality. Given (V,ℬ,μ) for σ-finite
measure μ, let K^(μ)(A,B):=μ(A∩ B),
A,B∈ℬ_fin where ℬ_fin:={ A∈ℬ;μ(A)<∞}.
Set 

    ℋ(K^(μ)):=the RKHS of functions on ,

s.t. for all F∈ℋ(K^(μ)), 

    F(A)=⟨ F,K_A^(μ)⟩_ℋ(K^(μ))=⟨ F,μ(A∩·)⟩ _ℋ(K^(μ)).

Set 

    L(h)(A):=⟨ h,X_A^(μ)⟩=𝔼(hX_A^(μ)),   L^*(K_A^(μ))=X_A^(μ),

h∈ H:=L^2(Ω,𝒞,ℙ), X_A^(μ)=
the Gaussian process of K^(μ), where 

    X_f^(μ)=∫ f(x)dX_x^(μ), f∈ L^2(μ).


A new operator ξ:=ξ^(μ):H⟶ L^2(μ),
where H:=L^2(Ω,𝒞,ℙ), and L^2(μ):=L^2(V,ℬ,μ). 

With the setting as above, ξ^(h)(h)∈ L^2(μ),
∀ h∈ H, is determined uniquely by

    ∫ξ^(μ)(h)fdμ=𝔼(hX_f^(μ)), ∀ h∈ H, ∀ f∈ L^2(μ).




ξ^(h)(h) is determined from (<ref>)
and Reisz since 

    |𝔼(hX_f^(μ))|^2≤‖ h‖ _H^2∫_V|f|^2dμ.

So we only need to show the estimate (<ref>), but it follows
again from the Ito isometry, as follows: Let h∈ H, and f∈ L^2(μ),
then 

    |𝔼(hX_f^(μ))|^2   (Schwarz)≤   ‖ h‖ _L^2(ℙ)^2‖ X_f^(μ)‖_L^2(ℙ)
        =    𝔼(|h|^2)𝔼(|X_f^(μ)|^2)
        =    𝔼(|h|^2)=:‖ f‖ _L^2(μ)^2∫_V|f|^2dμ

where we used the Ito isometry in the last step. 


ξ^(μ) is contractive, 

    ‖ξ^(μ)(h)‖_L^2(μ)≤‖ h‖ _H, ∀ h∈ H.




The two operators I^(μ) and ξ^(μ)
are specified as follows: 

    L^2(μ)∋f[rr]_I^(μ)       X_f^(μ)@/_1.3pc/[ll]_(⋯)^*∈ H=L^2(Ω,𝒞,ℙ)


    L^2(μ)∋ξ^(μ)(h)@/^1.2pc/[rr]^(ξ^(μ))^*       h@/^1.2pc/[ll]^ξ^(μ)∈ H
    =L^2(Ω,𝒞,ℙ)

We have 

    ξ^(μ)=(f⟶ X_f^(μ))^*=the adjoint operator, and


    f⟶ X_f^(μ)=(ξ^(μ))^*, and


    f⟼ X_f^(μ)=I^(μ)(f) is isometric (Ito).

Moreover, 

    (I^(μ))^*I^(μ)=I_L^2(μ), while


    I^(μ)(I^(μ))^*=Q_μ=projection on ;

, proj on. 

Or, we may rewrite (<ref>)-(<ref>) as 

    ξ^(ξ)I^(μ)    =identity operator on 
    
    I^(μ)ξ^(μ)    = the proj in  onto the range of






§ PARSEVAL FRAMES IN THE MEASURE CATEGORY


Let U be a set, and let K:U× U⟶ℝ
be a positive definite (p.d.) kernel. We assume that the corresponding
RKHS ℋ(K) is separable. (The result below
will apply mutatis mutandis also to complex p.d. kernels K:U× U⟶ℂ,
but for simplicity, we shall state our theorem only in the real case.) 

Let (S,ℬ,μ) be a measure space with μ
assumed positive and σ-finite. 

We shall say that L^2(μ) is a feature
space if there is a function r:U⟶ L^2(μ)
such that 

    K(x,y)=∫_Sr_x(s)r_y(s)dμ(s), ∀ x,y∈ U.

(In the complex case, the RHS in <ref> will instead be ∫_Sr_x(s)r_y(s)dμ(s).
See also fs.)



The notion of feature space derives from the setting
of machine learning <cit.>,
where learning optimization is made precise with the use of a choice
of reproducing kernel Hilbert space (RKHS). In practical terms, a
choice of feature space refers to a specified collections of features
that used to characterize data. For example, feature space might be
(Gender, Height, Weight, Age). In a support vector machine (SVM),
we might want to consider a different set of characteristics in order
to describe such data as (Gender, Height, Weight) etc; and we will
then arrive at mappings into other feature spaces. In finite dimension,
we may have feature spaces referring to some fixed number n of dimensions.
Since data is “large” it is useful
to consider feature spaces to be function spaces, especially a choice
of L^2-spaces. The term feature space is used often in the machine
learning (ML) literature because a task in ML is feature extraction.
Hence we view all variables as features.



We always have two distinguished feature spaces in the L^2-category:
l^2(ℕ) vs Gaussian. 

CASE 1. S=ℕ (note the separability assumption.), and μ:=
the counting measure. If { h_n} _n∈ℕ
is such that 

    K(x,y)=∑_n=1^∞h_n(x)h_n(y), ∀ x,y∈ U;

then h_n∈ℋ(K) for all n∈ℕ,
and { h_n} _n∈ℕ is a Parseval
frame in ℋ(K), i.e., 

    ‖ F‖ _ℋ(K)^2=∑_n=1^∞|⟨ F,h_n⟩ _ℋ(K)|^2;

and 

    F(x)=∑_n=1^∞⟨ F,h_n⟩ _ℋ(K)h_n(x)

is also strongly convergent, for all x∈ U, and F∈ℋ(K). 



The lemma follows from standard RKHS theory <cit.>.
An important point is to note that if K and { h_n} _n∈ℕ
satisfy the assumptions in CASE 1, then h_n∈ℋ(K)
for ∀ n∈ℕ. We may get this as an application of
rk1. Indeed, let n_0∈ℕ be fixed. Let k∈ℕ,
{α_i} _1^k, { x_i} _1^k,
α_i∈ℝ, x_i∈ U; then 

    |∑ _iα_ih_n_0(x_i)|^2   ≤   ∑ _n∈ℕ|∑ _iα_ih_n(x_i)|^2
        =    ∑ _i∑ _jα_iα_j∑ _n∈ℕh_n(x_i)h_n(x_j)
       by =   ∑ _i∑ _jα_iα_jK(x_i,x_j);

and so the premise in rk1 holds, and we conclude that h_n_0∈ℋ(K). 

CASE 2. Set S=ℝ^U, ℬ= the cylinder σ-algebra,
and μ= the Gaussian probability measure on S determined by
its finite samples: k∈ℕ, { x_i} _1^k,
x_i∈ U. On ℝ^k, define the standard centered
Gaussian, so with mean 0, and covariance matrix (K(x_i,x_j))_i,j=1^k.
Then apply Kolmogorov consistency, and μ=ℙ_Kolm(K)
will be the corresponding measure, also called the Wiener measure.
Setting, for x∈ U, 

    r_x(s)=s(x);

and the desired conclusions follow: 


  * Each r_x∈ L^2(μ) is Gaussian,

  * 𝔼(r_x)=∫ r_xdμ=0, 

  * 𝔼(r_xr_y)=K(x,y), ∀ x,y∈ U. 



Let K:U× U⟶ℝ be a positive definite
(p.d.) kernel, and let (S,ℬ,μ,r) be a feature
space as specified in p1; in particular, we have r_x∈ L^2(μ)
, ∀ x∈ U, and K(x,y)=∫_Sr_x(s)r_y(s)dμ(s),
see (<ref>). 


  * Set 

    R(x,A)=∫_Ar_x(s)dμ(s),

for x∈ U, A∈ℬ_fin, where 

    ℬ_fin={ A∈ℬ;μ(A)<∞} ;

then R in (<ref>) is a measure in the second variable, and
it is measurable in x (∈ U). 

  * For all F∈ℋ(K), we have 

    ‖ F‖ _ℋ(K)^2=∫_S|⟨ F(·),R(·,ds)⟩ _ℋ(K)|^2,

and 

    F(x)=∫_Sr_x(s)⟨ F(·),R(·,ds)⟩ _ℋ(K).





We study the the parallel between the present conclusions (<ref>)-(<ref>),
and the more familiar ones (<ref>)-(<ref>) from standard
Parseval frame-theory, see, e.g., <cit.>,
and also see <cit.> for direct integrals.



Note first that there is a natural isometry J defined by limits
and closure as follows: 

    J(∑ _iα_iK(·,x_i))=∑ _iα_ir_x_i.

Indeed for finite sample k, α_i∈ℝ, x_i∈ U,
1≤ i≤ k, we have 

    ‖∑ _iα_iK(·,x_i)‖ _ℋ(K)^2=‖∑ _iα_ir_x_i‖ _L^2(μ)^2

since both sides in (<ref>) reduce to ∑_i∑_jα_iα_jK(x_i,x_j).
As a result, in order to verify (<ref>)-(<ref>), we
need only consider the case F(·)=K(·,y)
when y∈ U is fixed. Then it is enough to show that (<ref>)
holds, and (<ref>) will follow. 

Let y∈ U be fixed, and assume F(·)=K(·,y).
Then 

    LHS_(<ref>)    =K(y,y), and
        =∫_S|r_y(s)|^2dμ=K(y,y),

by (<ref>). Similarly, 

    LHS_(<ref>)    =K(x,y), and
    RHS_(<ref>)    =∫_Sr_y(s)r_x(s)dμ(s)=K(x,y),

again from an application of assumption (<ref>).




§ TRANSFORMS


Let K:U× U⟶ℝ be a positive kernel.
(We shall state the result below for the real case but extensions
to p.d. functions with values in ℂ are straightforward;
even to the case of operator valued kernels.) Now let (S,ℬ,μ)
be a measure space with μ fixed and assume σ-finite.
We shall further assume that L^2(μ) is a feature
space; see p1 and p1, i.e., we assume that there
is a function, UL^2(μ), F(x)=r_x(·)(∈ L^2(μ))
such that (<ref>) holds.

 


  * With the setting (K,U,μ,{ r_x} _x∈ U)
as above, there is then a unique isometry J:ℋ(K)⟶ L^2(μ)
specified by 

    J(K(·,x))=r_x.


  * The adjoint operator of J is L:=J^*:L^2(μ)⟶ℋ(K),
given by 

    (Lh)(x)=∫_Sh(s)r_x(s)dμ(s),

∀ h∈ L^2(μ), and x∈ U. 

  * Q:=JJ^*=L^*L is the projection in L^2(μ)
onto the closed subspace spanned by { r_x(·);x∈ U}(⊆ L^2(μ)). 




(<ref>) Let the setting be as specified in the Proposition.
Define J as in (<ref>) and extend by linearity, first on
finite linear combinations 

    J(∑ _iα_iK(·,x_i))=∑ _iα_ir_x_i(·).

It is isometric, since 

    ‖∑ _iα_iK(·,x_i)‖ _ℋ(K)^2=∑ _i∑ _jα_iα_jK(x_i,x_j),

and 

    ∫_S|∑ _iα_ir_x_i(·)|^2dμ    =∑ _i∑ _jα_iα_j∫_Sr_x_ir_x_jdμ
        =∑ _i∑ _jα_iα_jK(x_i,x_j),

where we used (<ref>) in the last step. Since J is thus
isometric on a dense subspace in ℋ(K), it
extends uniquely by limits, to define an isometry J:ℋ(K)⟶ L^2(μ)
as required in (<ref>).

(<ref>) We now turn to the operator L:L^2(μ)⟶ℋ(K)
as in (<ref>). The important point is that L maps into ℋ(K).
This follows from an application of rk1 as follows. Let
n∈ℕ, α_i∈ℝ, x_i∈ U, 1≤ i≤ n;
then 

    |∑ _iα_i(Lh)(x_i)|^2    =    |∫_S(∑ _iα_ir_x_i)hdμ|^2
       Schwarz≤   ∫_S|∑ _iα_ir_x_i|^2dμ∫_S|h|^2dμ
       by =   ∑ _i∑ _jα_iα_jK(x_i,x_j)‖ h‖ _L^2(μ)^2,

and the conclusion now follows from rk1.

(<ref>) We have, since Lh∈ℋ(K),

    ⟨ K(·,x),Lh⟩ _ℋ(K)    =    (Lh)(x)
       by =   ∫_Sr_xhdμ=⟨ r_x,h⟩ _L^2(μ),

so L^*(K(·,x))=r_x, L^*=J, and
J^*=L, now follow from (<ref>). Since J is isometric,
Q=JJ^* is the projection specified in (<ref>) in the
Proposition.




§ HILBERT SPACES OF SIGNED MEASURES, AND OF DISTRIBUTIONS


Let K:U× U⟶ℝ be a fixed positive definite
(p.d.) kernel, and let ℋ(K) be the corresponding
RKHS. Suppose U is a metric space, and that K is continuous.
(This is not a strict restriction since K automatically induces
a metric d_K on U given by 

    d_K(x,y)     =‖ K(·,x)-K(·,y)‖ _ℋ(K)
        =(K(x,x)+K(y,y)-2K(x,y))^1/2,

and 

    |K(x_1,y)-K(x_2,y)|≤ d_K(x_1,x_2)K(y,y)^1/2, ∀ x_1,x_2,y∈ U.)


Introduce the Dirac delta measures {δ_x},
x∈ U, we get δ_xKδ_y=K(x,y); or more
precisely, 

    ∫_U∫_UK(s,t)dδ_x(s)dδ_y(t)=K(x,y).

If n∈ℕ, α_i∈ℝ, x_i∈ U, 1≤ i≤ n,
set 

    ξ:=∑ _i=1^nα_iδ_x_i,

and we get 

    ξ Kξ=∑ _i∑ _jα_iα_jK(x_i,x_j)=‖∑ _iα_iK(·,x_i)‖ _ℋ(K)^2.

Hence, if we complete the measures from (<ref>) with respect
to (<ref>), we arrive at a Hilbert space ℒ(K)
consisting of signed measures, or of more general linear functionals,
e.g., distributions.

Let K, U, ℋ(K) and
ℒ(K) be specified as above; then 

    J(∑ _iα_iδ_i):=∑ _iα_iK(·,x_i)

defines an isometry of ℒ(K) onto ℋ(K)
. 



This follows from the definitions. In particular, for ξ∈ℒ(K),
we have, by (<ref>)

    ξ Kξ=‖ Jξ‖ _ℋ(K)^2.



Let K, U, ℋ(K) and ℒ(K)
be as stated in hm1, and let (S,ℬ,μ)
be a σ-finite measure space such that L^2(μ)
is a feature space; see p1. Then a signed measure ξ
is in ℒ(K) if and only if 

    ∫_S|∫_Ur_x(s)dξ(x)|^2dμ(s)<∞;

and in this case ξ Kξ= the RHS in (<ref>).



Since ℒ(K)ℋ(K)
and K(·,x)⟶ r_x extends by limiting
and closure to an isometry ℋ(K)⟶ L^2(μ),
by tr1, the following computation is valid when ξ∈ℒ(K),
and vise versa: 

    ∫_U∫_UK(x,y)dξ(x)dξ(y)     =∫_U∫_U∫_Sr_x(s)r_y(s)dμ(s)dξ(x)dξ(y)
        =∫_S|∫_Ur_x(s)dξ(x)|^2dμ(s)=RHS_(<ref>).

 


If U× Uℝ(or )
is C^∞, or analytic for suitable choices of Uand K,
then we may have distribution solutions ξ to ξ Kξ<∞.
A simple example illustrating this is U=(-1,1)= the
interval, and 

    K(x,y)=1/1-xy.

For n∈ℕ, let ξ=δ_0^(n)= the
n^th derivative of the Dirac measure at x=0. Then 

    ∫_UK(x,y)dξ(y)=n!x^n/(1-xy)^n+1|_y=0=n!x^n;

and

    ∫_U∫_UK(x,y)dξ(x)dξ(y)=(n!)^2.

In fact, 

    δ_0^(n)Kδ_0^(m)=δ_n,m(n!)^2, ∀ n,m∈ℕ;

so the system {δ_0^(n)}_n∈{ 0}∪ℕ
is orthogonal and total in ℒ(K). If x∈ U\{ 0},
then 

    δ_x=∑ _n=0^∞x^n/n!δ_0^(n),

as an identity for compactly supported distributions. 



The co-authors thank the following colleagues for helpful and enlightening
discussions: Professors Daniel Alpay, Sergii Bezuglyi, Ilwoo Cho,
Myung-Sin Song, Wayne Polyzou, and members in the Math Physics seminar
at The University of Iowa.

amsalpha



