Adaptive posterior contraction rates for empirical Bayesian drift estimation of a diffusion
Jan van Waaij
	

Due to their conjugate posteriors,
Gaussian process priors are attractive
 for estimating the drift of 
 stochastic differential equations
 with continuous time observations. 
 However, their performance strongly 
 depends on the choice of the hyper-parameters. 
 We employ the marginal maximum likelihood estimator to estimate the scaling and/or smoothness parameter(s) of the prior and show that the corresponding posterior has optimal rates of convergence. General theorems do not apply directly to this model as the usual test functions are with respect to a random Hellinger-type metric. 



We allow for continuous and discrete, one- and two-dimensional sets of hyper-parameters, where optimising over the two-dimensional set of smoothness and scaling hyper-parameters is shown to be beneficial in terms of the adaptive range. 





  §.§.§ Summary




We assume a continuous time observation X^T=(X_t:t∈[0,T]), T>0, which is a realisation of the stochastic differential equation (SDE) X_t=θ_0(X_t)dt+dW_t, with W_t a Brownian motion. We are interested in estimating θ_0, which is only assumed to be 1-periodic, zero-mean (∫_0^1θ_0(x)dx=0) and β-Sobolev smooth for some unknown β>0. We equip the space of 1-periodic L^2-functions (on [0,1]) with a Gaussian process prior, given by s∑_k=1^ Tk^-α-1/2Z_kϕ_k, where the hyper-parameters s and/or α are estimated from the data using the marginal maximum likelihood estimator. Inference is done with a posterior. Fixing s and optimising over α∈ A=[1/2+δ,√(log T)] (for a small fixed constant δ>0) leads to the optimal rate of posterior convergence T^-β/1+2β, as long as the smoothness β of the true parameter is at least 1/2+δ. Fixing α>1/2 and optimising over s∈ S(α)=T^-1/4+4α,T^α leads to optimal rates as long as 0<β≤α+1/2. Optimising over α∈ A and s∈ S(α) simultaneously leads to optimal rates for all β>0. 
The same results are obtained when A and S(α) are replaced with a finite discrete grid that is `dense' enough in A and S(α), respectively. 
For the proof we use the approach of <cit.> and        <cit.>, adapted to the SDE model. 

	

§ INTRODUCTION

	
 Nonparametric Bayesian procedures for stochastic differential equations (SDEs) have recently received significant attention, which is partly motivated by the many applications SDEs have in science and economics (e.g. <cit.>, <cit.>  climate research,  <cit.> neurobiology, and <cit.>, <cit.> finance), and the many fast numerical schemes that have been developed for Bayesian nonparametric procedures for SDEs, see <cit.>, <cit.>, <cit.>, <cit.> and <cit.>, for example, and the Yuima R-package <cit.> for statistical software and the Julia-packages <cit.> and  <cit.>. 

	This motivates the study of the 
	theoretical performance of these 
	nonparametric Bayesian 
	procedures. Posterior consistency 
	or posterior contraction rates for 
	nonparametric Bayesian models for 
	diffusions are examined in 
	<cit.>, 
	<cit.>, 
	<cit.>, 
	<cit.>, 
	<cit.>, 
	<cit.>, 
	<cit.>, and 
	<cit.>. To 
	the best of the author's knowledge, no 
	attention has been given to 
	empirical Bayes methods for 
	diffusions and their asymptotic 
	properties, which is the subject of 
	the current study. 
	
	We assume a given continuous time observation  X^T=X_t:t∈[0,T], T>0 that is a weak solution to the stochastic differential equation 
	
    dX_t=θ_0(X_t)dt+dW_t,

	where W_t is a Brownian motion (BM) and the unknown parameter θ_0 is a one-periodic real-valued function on  with ∫_0^1θ(x)dx=0  and ∫_0^1θ_0(x)^2dx<
	∞. Equivalently, θ_0 can be seen as a measurable square integrable function on the unit circle, with zero mean. The space of this functions is denoted by L^2(). 
	
	In molecular dynamics, <ref> (<cit.>) is used to model the angle between atoms in a molecule. <cit.> proposes a Gaussian process prior on the unknown drift and a numerical scheme to sample from the posterior is given. In <cit.>, posterior consistency is shown for this prior. In <cit.>, minimax posterior convergence rates are shown for the prior whose law is defined by the random function
	
    θ=s∑_k=1^∞ k^-1/2-αZ_kϕ_k,    Z_ki.i.d.∼N(0,1),

	for fixed positive s and α and when θ_0 is α-Sobolev smooth. In addition, if s is equipped with a specific (hyper)prior, then adaptation with minimax rates (up to a multiplicative constant) are shown as long as the Sobolev-smoothness of the true parameter is at most α+1/2, with fixed α. Posterior convergence rates are shown for every Sobolev smoothness when α is equipped with a certain hyperprior, and s is set to one. General conditions for posterior contraction for continuously observed diffusions with several examples are examined in <cit.>. Nonparametric Bayesian procedures with discrete observations of <ref> are considered in <cit.> (low frequency regime) and <cit.> (high frequency regime). Nonparametric drift estimation is studied from the frequentist perspective in <cit.>, <cit.>, <cit.>,  <cit.>, <cit.>, <cit.>, and  <cit.>, among others.  

	Going back to the Bayesian procedure, one quite often sees that the hyperprior destroys the appealing numerical properties of the prior as the hierarchical prior is no longer conjugate, for example. Therefore, in practise, one often relies on  empirical Bayes methods, in which the hyper-parameters of the prior are estimated from the data and then plugged into the prior. Inference is done via the corresponding posterior.
	
General methods to study posterior contraction for empirical Bayes procedures in different models with the marginal maximum likelihood estimator (MMLE or MML) as an estimator for the hyper-parameter(s) are examined in <cit.>, <cit.>, and <cit.>. We follow the approach of <cit.> and        <cit.>, the difference in the current study is that the tests to distinguish between the  corresponding drift functions are in the random Hellinger metric (as introduced in <cit.>), which is with high probability equivalent to the L^2-metric. 
 The basic idea is first to determine a `good set' of hyper-parameters that contain the  MMLE with high probability, 
 and that the corresponding posteriors associated with this good set converge uniformly with the desired 
 rate, under the true parameter. We consider  MML estimators of the smoothness and scaling parameters, as well as an MML estimator for two-dimensional hyper-parameters λ=(α,s ) simultaneously. Hyper-parameters in different 
 parts of the hyper-parameter space may lead to optimal rates; for example, for rough truths, small α and 
 moderate s, or larger α and larger s values. This may lead to large or `strange' formed sets 
 Λ_0 of good parameters. The general theorem of <cit.> 
 allows for two and higher dimensional hyper-parameters, but they do not provide an example of such a prior. To the best of our knowledge, we are the first to do so using their approach. 
However, we find that this does not lead to complications for the analysis, but it does improve the results as it 
leads to an optimal rate for every β-Sobolev smooth function for all β>0, where the (proven) adaptive 
range is smaller when we only optimise over α or s with the other hyper-parameter fixed. 





	
	

§ THE MARGINAL MAXIMUM LIKELIHOOD ESTIMATOR AND THE EMPIRICAL POSTERIOR

	
	For a family of priors Π_λ:λ∈Λ, we define the marginal likelihood as 
    ∫ p^θ(X^T)dΠ_λ(θ),

	where 
    p^θ (X^T)=∫_0^T θ(X_t)dX_t-1/2∫_0^Tθ(X_t)^2dt

	is the density 	relative to the Wiener measure. 
	
	The maximum marginal likelihood estimator (MMLE) is a λ̂ in Λ that maximises the marginal  likelihood, 
	
    λ̂=max_λ∈Λ∫ p^θ(X^T)dΠ_λ(θ).

	As such maximum may not exist, or is difficult to find exactly, we also allow any choice of λ̂ in Λ for which 
    ∫ p^θ(X^T)dΠ_λ̂(θ)≥1/2sup_λ∈Λ∫ p^θ(X^T)dΠ_λ(θ).

	Obviously, a λ̂ satisfying <ref> also satisfies <ref>, so we work with λ̂ satisfying <ref> from now on. The choice for the factor 1/2 is, of course, somewhat arbitrary, but is chosen to keep the analysis and notation simple.
	
	For fixed λ, the posterior of a measurable set A⊆ L^2() is given by
	
    Π_λ(A| X^T)=∫_A  p^θ(X^T)dΠ_λ(θ)/∫  p^θ(X^T)dΠ_λ(θ).

	The posterior is well defined, see <cit.>.
We study posterior convergence rates of Π_λ̂(| X^T)=Π_λ(| X^T)|_λ=λ̂.  We refer to Π_λ̂(| X^T) as the empirical posterior, and to Π_λ̂() as the empirical prior. 

The family of Gaussian priors we consider is displayed below, where {ϕ_k}_k=1^∞ is the orthogonal Fourier basis of L^2() defined by ϕ_2k-1(x)=√(2)sin(2π kx), ϕ_2k(x)=√(2)cos(2π k x),  k∈ 
 and Z_i:i∈ are independent standard normally distributed random variables. The prior Π_λ is defined as the law of the random process defined by 
    t↦ s∑_ k=1^ T k^-α -1/2Z_kϕ_k(t),

 where λ= (α, s) is an element of Λ, which is a subset of  
    Λ_1:=(α,s):1/2+δ≤α≤√(log T),   T^-1/4+4α≤ s≤  T^α,

 and where δ>0 is a small fixed constant. The condition that α is bounded away from 1/2 is found to be necessary in our approach to derive a test that distinguishes between drift functions. It would be worthwhile if future research could alleviate this condition. 
For ease of  notation, we suppress the dependence on T. 
	
	
We say that the posterior contracts with rate _T↓ 0, as T→∞, when for some constant G>0 
    Π_λ̂θ:θ-θ_0≤ G_T| X^T→ 1,

	in ^θ_0-probability as T→∞. 
	
	
	We obtain the following result: 
	
Let θ_0 be β-Sobolev smooth, that is, θ_0∈ L^2() is of the form θ_0=∑_ k=1^∞θ_kϕ_k and satisfies ∑_ k=1^∞ k^2βθ_k^2<∞. Then the posterior contracts with rate T^-β/1+2β, when

	
  * Λ = (α,s):T^-1/4+4α≤ s≤ T^α, for some fixed α>1/2, and β≤α+1/2, or,
	
  * Λ = (α,s): 1/2+δ≤α≤√(log T) for some fixed s>0, and when β≥ 1/2+δ, or,
	
  * Λ=Λ_1 and β>0.

The same results hold when Λ in each of the three cases is replaced by the discrete sets 

  * Λ=(α, lT^-1/4+4α):l∈, 1≤ l≤ T^α+1/4+4α, for some fixed α>1/2

  * Λ=(k/log T, s):k∈,(1/2+δ)log T≤ k≤ (log T)^3/2, for some fixed s>0,

  * Λ={(k/log T, lT^-1/4+4k/log T):k,l∈,(1/2+δ)log T≤ k≤ (log T)^3/2,
Λ={(k/log T, lT^-1/4+4k/log T):k,l∈,(1/2+δ) 
1≤ l≤ T^k/log T+1/4+4k/log T},
 
respectively.   
Note that similar results are obtained in <cit.> 
for a hyperprior on the scaling (case 1) and a hyper-parameter on the 
smoothness parameter (case 2), with the difference that in the latter case adaptivity for all 
β>0 is obtained (<cit.>), which, in our study, is only 
obtained when α and s  are estimated at the same time.
		
	

§ THE PROOF OF <REF>

	
	 
	Using the equivalence of the measures P^θ, for fixed T (see <cit.>), the posterior <ref> can be written as 
	
    Π_λ(A| X^T)=   ∫_A p^θ(X^T)/p^θ_0(X^T)dΠ_λ(θ)/∫ p^θ(X^T)/p^θ_0(X^T)dΠ_λ(θ)	,
    	andp̅^θ(X^T):=   p^θ(X^T)/p^θ_0(X^T)
    
    	=   ∫_0^T(θ(X_t)-θ_0(X_t))dW_t-1/2∫_0^T(θ(X_t)-θ_0(X_t))^2dt

	is the density of P^θ relative to P^θ_0.  
	
	  

	Since p̅^θ(X^T)=p^θ(X^T)/p^θ_0(X^T) differs from p^θ(X^T) only by a positive multiplicative constant not depending on θ or λ, the λ that satisfies <ref> satisfies also  <ref> with p^θ replaced by p̅^θ.
	
	
	For technical reasons, we introduce a positive constant K (which does not depend on λ or T, but may depend on θ_0). It is easily shown that for some  unique _λ (dependent on K, T and θ_0)
	
	
    Π_λ(θ-θ_0_2<K_λ)=e^-T_λ^2.

	Let _0=inf_λ∈Λ_λ. Intuitively, we can think of the prior Π_λ that minimises _λ as one that most favours the true parameter out of all priors, by setting the maximum prior mass around θ_0 
	and is therefore expected to give the best possible rates of posterior convergence.
	
	For a constant M≥ 2, we define 
	
	
    Λ_0=λ∈Λ:_λ≤ M_0

	as the set of all  λ∈Λ where the prior Π_λ puts a significant amount of prior mass around θ_0. Note that this set is non-random but does depend on the unknown true parameter θ_0.
	In the next section, <ref>, we prove the existence of a sequence of events F (depending on T), which asymptotically have ^θ_0-probability one, and on which λ̂∈Λ_0. Obviously, for a constant H>1, we have
	
    _θ_0[Π_λ̂(θ-θ_0_2≥ HM_0  | X^T)]
    		≤_θ_0[sup_λ∈Λ_0Π_λ(θ-θ_0_2≥ HM_0  | X^T)_ F]+^θ_0(F^c).

	
	In <ref> we show that for sufficiently large H, 
    _θ_0[sup_λ∈Λ_0Π_λ(θ-θ_0_2≥ HM_0  | X^T)_F]→0

	as T→∞. For each respective choice of Λ in <ref>, the upper bounds for _0 in <ref> then allows us to conclude the theorem. 
	
	

§ THE ASYMPTOTIC BEHAVIOUR OF THE MARGINAL MAXIMUM LIKELIHOOD ESTIMATOR
 
	
	In <cit.>, it is shown that for a high probability event the Hellinger distance 
	
    h(θ,θ')= √(∫_0^T(θ(X_u)-θ'(X_u))^2du)

	and the square integrable (L^2-) norm is equivalent. To be precise, for any constants 0<c_ρ<1<C_ρ satisfying 
	
	
    c_ρ^2 <   inf_x∈[0,1]ρ(x)≤sup_x∈[0,1]ρ(x)<C_ρ^2
    		withρ(x)=   2∫_0^xθ_0(y)dy/∫_0^12∫_0^yθ_0(z)dzdy

	the P^θ_0-probability of the event 
	
    E=c_ρ√(T)θ-θ'_2≤ h(θ,θ')≤ C_ρ√(T)θ-θ'_2,∀θ,θ'∈ L^2()

converges to 1, as T→∞. 
	 
	 
	 Let A<B be positive constants. Define the event (which depends on T)  
	 
    F:=    E∩sup_λ∈Λ_0^c∫p̅^θ(X^T)dΠ_λ(θ)≤ e^-BT_0^2∩sup_λ∈Λ_0∫p̅^θ(X^T)dΠ_λ(θ)≥ e^-AT_0^2,

	 where Λ_0^c=ΛΛ_0 is the complement of Λ_0 in Λ. 
	 
	 As T_0^2→∞ (see <ref>) and A<B,  it follows that for large T, in the event F,
	 
    ∫p̅^θ(X^T)dΠ_λ̂(θ)≥1/2sup_λ∈Λ∫p̅^θ(X^T)dΠ_λ'(θ)
    	 ≥1/2sup_λ∈Λ_0∫p̅^θ(X^T)dΠ_λ(θ)≥1/2e^-AT_0^2> e^-BT_0^2.

	Which implies that for large T, in the event F,  the estimator λ̂ is in the set of good hyper-parameters Λ_0.

	 
	 	There are constants 0<A<B, so that when M in <ref> is sufficiently large, P^θ_0(F)→1 as T→∞. Moreover, for every λ'∈Λ_0 satisfying _λ≤2_0, we have in the event F 
    ∫p̅^θ(X^T)dΠ_λ'(θ)≥ e^-AT_0^2.

	 
	 
	 
	 	 
	 	 By the construction of Λ_0, there is a λ'∈Λ_0 with 
	 	_λ'≤ 2_0 (and clearly _λ'≥_0). Using <cit.> and  _0≤_λ'≤ 2_0, we see that for every such λ',
	 	
	 	
    P^θ_0∫p̅^θ(X^T)dΠ_λ'(θ)>e^-4(C_ρ^2K^2+1)T_0^2∩ E≥  P^θ_0(E)--1/8C_ρ^2T_0^2.

	 	
	 	As ^θ_0(E) converges to one, and the other term to zero, by  <ref>, it follows that the quantity on the left of the inequality converges to one. 
	 	
	 	
	 	Let B>4(C_ρ^2K^2+1) be a constant. Let us now concentrate our attention on the event 
	 	
	 	
    E∩sup_λ∈Λ_0^c∫p̅^θ(X^T)dΠ_λ(θ) > e^-BT_0^2.

	 	
	 	We have to show that the P^θ_0-probability of    <ref> converges to zero. We have
    sup_λ∈Λ_0^c∫p̅^θ(X^T)dΠ_λ(θ)
    	 		≤max_h∈1,…,Nsup_λ∈ I_h∫p̅^θ(X^T)dΠ_λ(θ)

	 	where N∈, I_h⊆Λ, h∈1,…,N so that ⋃_k=1^N I_h⊇Λ_0^c, choose λ_1=(α_1,s_1),…,λ_N=(α_N,s_N)∈Λ_0^c and sets I_1,…, I_N⊆Λ, so that α_h=infα:(α,s)∈ I_h, and 
diam(α:(α,s)∈ I_h)≤ T^-1.5√(log T)-1.5/log T and |s-s_h|≤ T^-1.5√(log T)-10/6 for all s so that (α,s)∈ I_h. The first and the second condition are trivially satisfied for case 1 and case 2 in <ref>, respectively. Note that if we chose a discrete set Λ,  we can choose singleton sets I_h=λ_h, λ_h∈Λ_0^c.
	 	Whatever set Λ  we chose,  N is bounded by e^5(log T)^3/2, 
	 	for sufficiently large T. It follows that
	 	
    P^θ_0[sup_λ∈Λ_0^c∫p̅^θ(X^T)dΠ_λ(θ)>e^-BT_0^2∩  E]
    ≤    e^5(log T)^3/2max_1≤ k≤ NP^θ_0[sup_λ∈ I_h∫p̅^θ(X^T)dΠ_λ(θ)>e^-BT_0^2∩  E ].

	 	
	 	
	 	It is only left to show that
    max_1≤ k≤ NP^θ_0[sup_λ∈ I_h∫p̅^θ(X^T)dΠ_λ(θ)>e^-BT_0^2∩ E ]=e^-5(log T)^3/2  as   T→∞.

	 	
	 	
	 	Let h∈1,…,N. Then λ_h∈Λ_0^c and so _λ_h≥ M_0. Let φ be the test function of <ref>, with =_λ_h and U=K and Θ the corresponding sieve, where K is the constant in <ref>, which can be chosen independent of h∈1,…,N. 
	 	
	 		
	 		
	 	For each  h∈1,…,N and for all λ∈ I_h, we define a 
	 	measurable map (which we call a transformation) 
	 	Φ_λ:ϕ_k:1≤ k≤ T (=Π_λ_h) 
	 	→ϕ_k:1≤ k≤ T (=Π_λ ) such that when 
	 	θ∼Π_λ_h, then Φ_λ(θ)∼Π_λ. Obviously, when we chose a discrete set Λ then 
	 	Φ_λ is the identity map. 
	 	 For θ∈ϕ_k:1≤ k≤ T, we have 
	 	 θ=∑_k=1^ Tθ_kϕ_k, for some θ_1,…,θ_ T∈. Define for 
	 	 λ=(α,s), 
    Φ_λ(θ)=s/s_h∑_k=1^ T 
    	 		k^α_h-αθ_kϕ_k.
 
	 	Φ_λ clearly satisfies the desired properties.
	 	Using this for a non-empty family of positive random variables,
	 	 Y_γ, γ∈Γ, sup_γ∈Γ Y_γ≤sup_γ∈ΓY_γ
	 	  and the transformation Φ_λ, we obtain 
	 		 	
	 	
    P^θ_0[sup_λ∈ I_h∫p̅^θ(X^T)dΠ_λ(θ)>e^-BT_0^2∩ E ]
    ≤   _θ_0φ
    	 		+_θ_0[∫sup_λ∈ I_hp̅^Φ_λ(θ)(X^T)dΠ_λ_h(θ)>e^-BT_0^2(1-φ)_E].
 
	 	The first term _θ_0φ≤ e^-C_1K^2T_λ_h^2≤ e^-C_1K^2M^2T_0^2≤ e^-M^2T_0^2, thereby using _λ_h≥ M_0 and choosing sufficiently large K, which can and will be done uniformly over h∈1,…,N, hereby noting that C_1 does not depend on K and on the h. With the help of the Markov inequality and Fubini's theorem, the second term is bounded by 
	 	
    e^BT_0^2∫_θ_0[sup_λ∈ I_hp̅^Φ_λ(θ)(X^T)(1-φ(X^T))_E]dΠ_λ_h(θ)
    
    	 		=e^BT_0^2∫_θ[sup_λ∈ I_h(p̅^Φ_λ(θ)(X^T)/p̅^θ(X^T))(1-φ(X^T))_E]dΠ_λ_h(θ)

	 	With the help of the Cauchy-Schwartz inequality and using  (1-φ(X^T))^2≤ 1-φ (X^T) (because 1-φ takes values in [0,1]), we find that <ref> is bounded by 
	 	
    e^BT_0^2√(∫_θ[sup_λ∈ I_h(p̅^Φ_λ(θ)(X^T)/p̅^θ(X^T))^2_E]dΠ_λ_h(θ))√(∫_θ[(1-φ(X^T))_E]dΠ_λ_h(θ)).

	 	
	 		For the integral in the second square root of the last expression we have the following bound:
	 		
	 		
    ∫_θ[(1-φ(X^T))_E]dΠ_λ_h(θ)
    	 			≤Π_λ_h[θ∈ L^2():θ-θ_0 < K_λ_h]
       +∫_θ∈Θ(λ_h):θ-θ_0≥ K_λ_h_θ[(1-φ(X^T))_E]dΠ_λ_h(θ)
    	 			+Π_λ_h( L^2()Θ(λ_h))
    ≤    e^-T_λ_h^2+e^-C_2KT_λ_h^2+e^-KT_λ_h^2≤  3 e^-M^2T_0^2,

	 		for sufficiently large K (note that K does also not depend on C_2) and using _λ_h≥ M_0.
	 		
	 		Let us now consider the other integral in <ref>.  
	 		In the cases in which we choose a discrete set, the left square root of <ref> is simply one.
	 		When we chose a continuous interval, we first bound the inner expectation uniformly over θ∈ L^2() with θ_2≤ D. 
	  theoremrestatabletheoremupperboundforQinalphacase
	 A constant T_0>0 exists, only depending on δ in <ref>, so that for every D>0 and θ∈ L^2() with θ_2≤ D and T≥ T_0,  
	
    _θ[sup_λ∈ I_hp̅^Φ_λ(θ)/p̅^θ(X^T)^2_E ]
    	≤ 12T + 8e^24C_ρ^2D^2T^-3√(log T)


The proof of this theorem is in <ref>.

Next, we need a bound on the `tail' of the prior, which is given in 

 For x/s≥1/√(2α),  we have 
    Π_λ (θ_2>x)≤   -α x^2/(s^2).



	Let Φ be the cumulative distribution function of a standard normally distributed random variable.  
It follows from <cit.> and elementary bounds on Φ that when x≥1/√(2α),

    ∑_k=1^ T k^-α-1/2Z_kϕ_k_2>x
    = ∑_k=1^ T k^-2α-1Z_k^2>x^2
    ≤∑_k=1^∞ k^-2α-1Z_k^2>x^2≤  1-Φ(√(2α) x)≤-α x^2.
 
This implies the result after substituting x/s for x.


Note that for Λ_1, x/s≥1/√(2α) when x ≥1/√(1+2δ)T^√(log T). 
As Π_λ(θ_2>0)=e^0, we may write
	
    ∫_θ[sup_λ∈ I_h(p̅^Φ_λ(θ)(X^T)/p̅^θ(X^T))^2_ E]dΠ_λ_h(θ)
    
    ≤   ∑_m=0^∞sup_θ≤√(m+1/1+2δ)  T^√(log T) P^θ[sup_λ∈ I_h(p̅^Φ_λ(θ)(X^T)/p̅^θ(X^T))^2_E]Π_λ_hθ_2≥√(m/1+2δ) T^√(log T)
    
    ≤   ∑_m=0^∞12T+824C_ρ^2m+1/1+2δT^-√(log T) e^- m/2
    =   12T/1-e^-1/2 + 8 24C_ρ^2/1+2δ T^-√(log T)
       + 8∑_m=1^∞- m T^-√(log T)1/2T^√(log T) - 24C_ρ^2/1+2δm+1/m
    
    ≤     31T +  22 + 29    
    ≤  32T,

for sufficiently large T. 
In summary, 

    ^θ_0sup_λ∈Λ_0^c∫p̅^θ(X^T)dΠ_λ(θ) > e^-BT_0^2∩  E≤   e^5(log T)^3/2e^-M^2T_0^2+√(96T)e^(B-M^2/2)T_0^2.


Using T_0^2≳ e^1/2√(log T) (<ref>),  which is eventually larger than 5(log T)^3/2, it follows that for M>√(2B), <ref> converges to zero, which completes the proof of the theorem.
	
	

§ POSTERIOR CONVERGENCE ON THE FAVOURABLE EVENT

	
	Let F be the event and M be the constant of <ref>. 		For some constant H>1, 
			 
    _θ_0[Π_λ̂(θ-θ_0_2≥ HM_0| X^T)_F]

	 converges to zero, as T→∞. 
	
	
	Choose λ'∈Λ_0 satisfying _λ'≤ 2_0. 
	It follows from ∫p̅^θ(X^T)dΠ_λ̂(θ)≥1/2∫p̅^θ(X^T)dΠ_λ'(θ) and <ref> that in the event F (with A defined in <ref>),
	
    Π_λ̂(θ-θ_0_2≥ HM_0| X^T)
    ≤ 	2	e^AT_0^2∫_θ-θ_0_2≥ HM_0p̅^θ(X^T)dΠ_λ̂(θ).

	There are subsets I_h', k=1,…,N' of Λ, containing a point λ_h'∈Λ_0, which is defined as in the proof of  <ref> with Λ_0^c replaced by Λ_0 and where N' satisfies the same upper bound as N, such that 
		Λ_0⊆⋃_k=1^N'I_h'.
	Let Φ_λ' be defined  as Φ_λ in the proof of  <ref>, with λ_h replaced by λ_h' and I_h replaced by I_h'. 
	Using M_0≥_λ_h', for every h∈1,…,N', we can bound for a constant A'>A, 
	
    P^θ_0[∫_θ-θ_0_2≥ HM_0p̅^θ(X^T)dΠ_λ̂(θ)>e^-A'T_0^2,F]
    ≤    N'·max_h∈1,…,N'P^θ_0[∫_θ-θ_0≥ H_λ_h'sup_λ∈ I_hp̅^Φ_λ'(θ)(X^T)dΠ_λ_h(θ)>e^-A'T_0^2,F].

	Let h∈1,…,N'. Let φ be the test functions of <ref> with λ=λ_h', =_λ_h and U=H≥ K. Then for  sufficiently large T, 		
	 		
	 		
	 
    P^θ_0[∫_θ-θ_0≥ H_s_h'sup_λ∈ I_h'p̅^Φ_λ'(θ)(X^T)dΠ_λ_h'(θ)>e^-A'T_0^2,F]
    ≤   _θ_0φ(X^T)
    	 	+e^A'T_0^2∫_θ-θ_0≥ H_λ_h'_θ_0[sup_λ∈ I_h'p̅^Φ_λ'(θ)(X^T)(1-φ(X^T))_F]dΠ_λ_h(θ)
    ≤    e^-H^2T_0^2+ e^A'T_0^2√(19Te^-C_2H^2_0^2+  e^-H^2T_0^2),

in a similar way as in the proof of <ref> and using that _λ≥_0.  Hence, for H>√(2A'/C_2∧1), Π_λ̂(θ-θ_0_2≥ HM_0| X^T)→ 0, as T→∞, in a similar way as in the proof of <ref>.
	 





§ THE PROOF OF <REF>

*

Note that W is a BM under P^θ_0, and so 

    W^θ_t:=W_t-∫_0^t(θ(X_u)-θ_0(X_u))du

is a BM under P^θ.
We may write 

    p̅^Φ_λ(θ)
    = p̅^θ (X^T)
     ∫_0^T(Φ_λ(θ)(X_u)-θ(X_u))dW_u^θ-1/2∫_0^T(Φ_λ(θ)(X_u)- θ(X_u))^2du.

Note that for all G>0

    _θ[sup_λ∈ I_hp̅^Φ_λ(θ)/p̅^θ(X^T)^2_E ]
    ≤ e^2G+∑_g=G^∞ e^2g+2^θ(sup_λ∈ I_hp̅^Φ_λ(θ)/p̅^θ(X^T)>e^g∩  E).

Let D>0 and θ∈  L^2() with θ_2≤ D. Note that in the event E, 

    ∫_0^(Φ_λ(θ)(X_t)-θ(X_t))dW_t^θ_T
    =  ∫_0^T(Φ_λ(θ)(X_t)-θ(X_t))^2dt
    ≤  C_ρ^2 T Φ_λ(θ)-θ_2^2.

We have 
    Φ_(α,s)(θ)-θ_2^2
    =∑_k=1^ Ts/s_hk^α_h-α-1^2θ,ϕ_k^2.

We next bound s/s_hk^α_h-α-1.  For this, we use the following

	For μ>0 and x≥0 we have 1-μ e^-x≤μ x + 1-μ.
	Moreover, for all μ>0 and x≥ 0, we have |1-μ e^-x|≤μ x + |1-μ|.


	Let us consider the first inequality. 
	Denote f(x)=1-μ e^-x and g(x)=μ x + 1-μ. Note that f(0)=g(0) and g'(x)=μ≥ f'(x)=μ e^-x, for all x≥ 0. It follows that g(x)≥ f(x) for all x≥0. This proves the first inequality. 
	
	For the second inequality, we note that when μ>e^x, |1-μ e^-x|=μ e^-x-1≤μ-1≤μ x + |1-μ|. When μ≤ e^x, we have |1-μ e^-x|=1-μ e^-x≤μ x + 1-μ≤μ x + |1-μ|. The proof is now complete.

Using lemma <ref>, and the fact that α_h≤α, we see that 
    |1-s/s_hk^α_h-α|
    ≤s/s_h|α-α_h|log k+|s-s_h|/s_h≤s/s_h|α-α_h|log T+|s-s_h|/s_h.
 
Using the lower bound for s∈Λ and the upper bound for |s-s_h| on I_h, we derive
    |s-s_h|/s_h≤  T^1/4+4αT^-1.5√(log T)-10/6≤ T^1/6+4δ-1/6 T^-1.5√(log T)-1.5≤ 0.5T^-1.5√(log T)-1.5,

for sufficiently large T. 
We also have 
    s/s_h≤   1+|s-s_h|/s_h≤ 1.5,

for sufficiently large T. 
It follows that for all λ=(α,s) and all k that we consider, using the upper bound on |α_h-α| on I_h, 
    |1-s/s_hk^α_h-α|≤  2T^-1.5√(log T)-1.5,

for sufficiently large T.

By the Cauchy inequality and the assumption on θ, we have |θ, ϕ_k|≤θ_2≤ D. 

Using the Bernstein inequality (<cit.>), for g>0, we obtain 

    ^θ[sup_λ∈ I_hp̅^Φ_λ(θ)/p̅^θ(X^T)>e^g, E]
    ≤sup_λ∈ I_h∫_0^T (Φ_λ(θ)(X_t)- θ(X_t))dW_t^θ > g, E
    ≤   Tsup_(α,s)∈ I_hmax_k∈1,…,Ts/s_hk^α_h-α-1θ,ϕ_k∫_0^Tϕ_k(X_t)dW_t^θ > g
    , E
    
    ≤    T max_k∈1,…,T∫_0^T ϕ_k(X_t)dW_t^θ > g/2DT^1.5√(log T)+0.5 , ∫_0^ϕ_k(X_t)dW_t^θ_T≤ C_ρ^2 T  
    ≤    T-g^2/4C_ρ^2D^2T^3√(log T).



Note that for g>0, Ag^2 - 2g ≥ g    
g ≥ 3A^-1.
It follows that for integers G≥ 12C_ρ^2D^2T^-3√(log T), 
    _θ[sup_s∈ I_hp̅^sθ/s_h(X^T)/p̅^θ(X^T)^2, E ]
    ≤ e^2G+e^2T∑_g=G^∞ e^-g
    = e^2G+e^2Te^-G/1-e^-1≤  e^2G+ 12T,

as e^2e^-G/1-e^-1≤e^2/1-e^-1≤ 12. 
Taking  G= 12C_ρ^2D^2T^-3√(log T), and using x≤ x+1,  for every real number x and e^2≤ 8, the last display is further bounded by 12T + 8e^24C_ρ^2D^2T^-3√(log T).






§ UPPER AND LOWER BOUNDS FOR _Λ
 
Let Π_λ^∞ be the distribution on L^2() defined by the law of
    s∑_ k=1^∞ k^-α-1/2Z_kϕ_k.


	It follows from <cit.> and <cit.> that for sufficiently small /s>0, β≤α+1/2 and 
	θ_0∈Ḣ^β(), 
	
    -C_1C_2 α (s/)^1/α+1/s^2^2β-2α-1/β≤Π_λ^∞(θ-θ_0_2<) ≤-C_3 α  (s/)^1/α.
 
	As in the proof of <ref>, we have 
    ∑_k=1^ T k^-1/2-αZ_kϕ_k_2<≥∑_k=1^∞ k^-1/2-αZ_kϕ_k_2<.

	
	For an inequality in the reverse order, we need the following modification of Markov's inequality
	
	 Let r>0, X be a non-negative random variable, and A be an event, such that A and X are independent. Then (A∩X≥ r) ≤1/r(A) X.
	

	The lemma follows from (A∩X≥ r)= _A _X≥ r≤_A X/r= 1/r(A) X, by independence.

	
		Note that 
	
    ∑_k= T + 1^∞ k^-1/2-αZ_kϕ_k_2^2 =  ∑_k= T + 1^∞ k^-1-2α≤1/2α T^-2α.

	Hence for >1/√(α)T^-α, an application of <ref> gives, 
    ∑_k=1^ T k^-1/2-αZ_kϕ_k_2<
    	=  ∑_k=1^ T k^-1/2-αZ_kϕ_k_2<, ∑_k= T +1^∞ k^-1/2-αZ_kϕ_k_2<
       +∑_k=1^ T k^-1/2-αZ_kϕ_k_2<, ∑_k= T +1^∞ k^-1/2-αZ_kϕ_k_2≥
    ≤   ∑_k=1^∞ k^-1/2-αZ_kϕ_k_2<2
    	+^-21/2αT^-2α∑_k=1^ T k^-1/2-αZ_kϕ_k_2<
    ≤   ∑_k=1^∞ k^-1/2-αZ_kϕ_k_2<2
    	+1/2∑_k=1^ T k^-1/2-αZ_kϕ_k_2<Rearranging the terms gives   ∑_k=1^ T k^-1/2-αZ_kϕ_k_2<≤   2∑_k=1^∞ k^-1/2-αZ_kϕ_k_2<2.
	
	
	Hence, for sufficiently small /s>1/√(α)T^-α, β≤α+1/2, and θ_0∈Ḣ^β(), 
	
    -C_1C_2α (s/)^1/α+1/s^2^2β-2α-1/β≤Π_λ(θ-θ_0_2<)  ≤ 2-C_3α (s/(2))^1/α.
 
	
	We have 
	
    2-C_3α (s/(2K_λ))^1/α≥Π_λ (θ-θ_0_2<K_λ)=e^-T_λ^2.

	It follows that for sufficiently small _λ/s 
	
    _λ≥  2^-1+α/1+2αC_3^α/1+2αK^-1/1+2αα^α/1+2αT^-α/1+2αs^1/1+2α.

	Using this lower bound we derive the lower bound 
    inf_λ∈Λ_1 T_λ^2
    	≥C̃_3 K^-1e^1/2√(log T),

	where C̃_3 is a constant.
Using the other bound of <ref>,  we obtain	
    -C_1C_2α (s/(K_λ))^1/α+1/s^2(K_λ)^2β-2α-1/β≤Π(θ-θ_0_2<K_λ)
    		=  e^-T_λ^2.

	
	Hence, either C_1C_2α (s/(K_λ))^1/α≥ T_λ^2/2 or C_11/s^2(K_λ)^2β-2α-1/β≥ T_λ^2/2. It follows that  

    _λ≤ 2^α/1+2αC_1^α/1+2αC_2^α/1+2αK^-1/1+2αα ^α/1+2α s^1/1+2αT^-α/1+2α⋁ 2^β/1+2αC_1^β/1+2αK^2β-2α-1/1+2αs^-2β/1+2αT^-β/1+2α.
	
		For each of the  three cases of <ref>, we derive an upper bound for _0 separately.
	
	
		
  * In case 1, α is fixed, so 
    _λ≲ s^1/1+2αT^-α/1+2α⋁ s^-2β/1+2α T^-β/1+2α.

		We see that the best possible upper bound for _0 (up to a constant) is attained when  
		
    s^1/1+2α T^-α/1+2α≍ s^-2β/1+2αT^-β/1+2α,

		that is, when s≍ T^α-β/1+2β.  Note that  for β∈ (0,α+1/2] a quantity proportional to T^α-β/1+2β is in Λ (also in the discrete case), and hence,  _0≲ T^-β/1+2β. 
		
  * In case 2, s is fixed, so 
    _λ≲α^α/1+2α T^-α/1+2α⋁ T^-β/1+2α.

		We see that the best possible upper bound for _0 (up to a constant) is attained when α=β+𝒪(1/log T). For sufficiently large T, Λ contains such an element, and in the discrete case, when β≥α+δ, in which case _0≲ T^-β/1+2β. 
		
  * In case 3, we have 
    _λ≲α^α/1+2α s^-1/1+2αT^-α/1+2α⋁ s^-2β/1+2αT^-β/1+2α.
 We see that the best possible upper bound for _0 (up to a constant) is attained when  α≥ (β-1/2)∨ (1/2+δ) and 
		
    T^-α/1+2αs^1/1+2α≍ s^-2β/1+2αT^-β/1+2α,

		that is, when s≍ T^α-β/1+2β. Such a pair (α,s) is in Λ (for sufficiently large T), and in the discrete case, and we have _0≲ T^-β/1+2β. 
	
	


§ THE EXISTENCE OF TEST FUNCTIONS



	
	There are positive constants C_1,C_2 and K, only depending on c_ρ and C_ρ, such that for every λ∈Λ and U≥ K, there are measurable sets (sieves)  Θ_λ⊆ L^2() satisfying
	
    Π_λ( L^2()Θ_λ)≤ e^-U^2T_λ^2,

	and  measurable maps φ_λ:C[0,T]→0,1, which satisfy
	
    _θ_0φ_λ(X^T) ≤    e^-C_1U^2T_λ^2,and for all θ∈Θ_λ,θ-θ_0_2≥ U_λ,_θ[(1-φ_λ(X^T))_E]≤    e^-C_2U^2T_λ^2.


 In this proof, let N(,S,d) denote the minimal number of balls of size  in a d-metric needed to cover a set S. The log of this number is referred to as the entropy. It follows from <cit.> and the equivalence of the h- and the L^2-metric on E (<ref>) that there are constants C_1, C_2, and c>0, only depending on c_ρ and C_ρ, such that when Θ_λ is a measurable set satisfying the entropy condition 
    log Nc_ρ_λ/8C_ρ,θ∈Θ_λ:θ-θ_0_2≤_λ,_2≤ c^2U^2T_λ^2,

	for U≥ K, then there are measurable maps φ_λ (depending on λ,T and U) taking values in {0,1} which satisfy <ref>.


We continue by showing that  such Θ_λ actually exist and also satisfy <ref>. For this, we follow <cit.>. Every θ∈ L^2() has an expansion θ=∑_k=1^∞θ_kϕ_k in the chosen orthonormal basis of L^2(). Recall that in the aforementioned study,  H_1^α+1/2 is the set of θ∈ L^2() for which ∑_k=1^∞ k^2α+1θ_k^2≤ 1 and L^2_1() is the closed unit ball in L^2(). We take the following sieves,	
    Θ_λ=R H_1^α+1/2()+c_ρ_λ/16C_ρ L^2_1().

Then, for some constant C>0 (only depending on λ,c_ρ and C_ρ), 
	
    Π_s( L^2()Θ_λ)≤-1/2R/s-√(C(s/_λ)^1/α)^2.

	Let 
	
    R    =sU√(2T)_λ+√(C(s/_λ)^1/α),

	then Π_s( L^2()Θ_λ)≤ e^-U^2T_λ^2, hence <ref> is satisfied. 
	For some constant Ĉ>0 (depending on λ,c_ρ, and C_ρ), the entropy is bounded as follows 
	
    log Nc_ρ_ s/8C_ρ,θ∈Θ_ s:θ-θ_0_2≤_ s,_2≤ĈR/_λ^2/(1+2α).

	For some constant C̃>0, this is bounded by
	
    C̃s^2/1+2αU^2/1+2αT^1/1+2α⋁ s^1/α_λ^-1/α
    
    	=    C̃ T_λ^2s^2/1+2αU^2/1+2αT^-2α/1+2α_λ^-2⋁ T^-1s^1/α_λ^-1+2α/α

	 Inserting the lower bound of <ref> shows that this is bounded for some constant C̅>0, (only depending on c_ρ,C_ρ, and λ) by 	 
    C̅ T_λ^2(UK)^2/1+2α⋁ K^1/α.

	 
	 
	 This is for U≥ K bounded by 
    C̅K^2-4α/1+2αU^2T_λ^2≤C̅K^-4δ/2+2δU^2T_λ^2≤  cU^2T_λ^2,

	  for sufficiently large K, which establish <ref>. 


	
	

§ DISCUSSION

	
	
	We examined posterior contraction rates for priors whose hyper-parameter(s) are estimated using the marginal maximum likelihood estimator. It is shown that estimating the smoothness and scaling parameter at the same time is advantageous in terms of the guaranteed adaptive range.
	
	 It would be nice to investigate ways to work around the requirement that α is bounded away from a half, which seems artificial to us, because in hierarchical Bayes one obtains adaptivity to every β>0 by putting a specific prior on α, which is supported on [0, log T], see <cit.>. We do need the condition  α≥ 1/2+δ to satisfy the entropy condition in the proof of the existence of tests, <ref>.  A similar phenomenon occurs in <cit.>, where they need α>1/2 and β>1/2 for the same prior in the log-linear model. This may be due to  the fact that in the chosen approach for  the proofs, one not only needs a lower bound on the amount of prior mass around the true parameter, as in hierarchical Bayes, but also an upper bound on the prior mass, see <ref> in this study and equations 2.1 and 2.2 in <cit.>.
	
	It may also be worthwhile investigating rates of convergence for the MLL estimator on the number of basis functions J, instead of using a fixed J= T basis functions, probably in combination with an estimator of the scaling parameter. In the hierarchical approach truncating the prior at a random J, via a hyperprior on J∈ (in combination with a prior on the scaling), gave both good asymptotic and numerical  results in <cit.> and <cit.>, respectively. 
	
	In <cit.>, the authors use  the infinite series <ref>, while we truncate at T. The reason is to control the uniform convergence over all posteriors in Λ_0. For this we use the bound <ref>, which grows to infinity when the number of basis functions grows to infinity. The advantage of <ref> is only theoretical, the prior with the law of <ref> does not depend on time, which makes the posterior a martingale. For case 1 with a continuous set Λ, and in cases 1, 2, and 3 with a discrete set in <ref>, one can use an infinite series prior, as |α-α_h|=0 in that case. 
	However, for simplicity, we decided to omit this. This is available from the author on request. In addition, due to limited computer memory one has to truncate in applications. 
 	
	


  
Acknowledgment

The author is grateful for the helpful comments from Harry van Zanten regarding an earlier version of this paper.


