A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems
Xuhui Meng George Em Karniadakis


































brown]Xuhui Meng
brown,PNNL]George Em Karniadakiscor


[cor]Corresponding author
[cor]george_karniadakis@brown.edu

[brown]Division of Applied Mathematics, Brown University, Providence, RI, USA, 02912
[PNNL]Pacific Northwest National Laboratory, Richland, WA, USA, 99354



Currently the training of neural networks relies on data of comparable accuracy
but in real applications only a very small set of high-fidelity data is available while inexpensive lower fidelity data may be plentiful. 
We propose a new composite neural network (NN) that can be trained based on multi-fidelity data. It is comprised of three NNs,
with the first NN trained using the low-fidelity data and coupled to two high-fidelity NNs, one with activation functions and another one without, in order to discover and exploit nonlinear and linear correlations, respectively, between the low-fidelity and the high-fidelity data. 
We first demonstrate the accuracy of the new multi-fidelity NN for approximating some standard benchmark functions but also a 20-dimensional function that is not easy to approximate with other methods, e.g. Gaussian process regression. Subsequently, we extend the  recently developed physics-informed neural networks (PINNs) to be trained with multi-fidelity data sets (MPINNs). MPINNs contain four  fully-connected neural networks, where the first one approximates the low-fidelity data, while the second and third construct the correlation between the low- and high-fidelity data and produce the multi-fidelity approximation, which is then used in the last NN that encodes the partial differential equations (PDEs). Specifically, in the two high-fidelity NNs a relaxation parameter is introduced, which can be optimized to combine the linear and nonlinear sub-networks. By optimizing this parameter, the present model is capable of learning both the linear and complex nonlinear correlations between the low- and high-fidelity data adaptively. By training the MPINNs, we can:  (1) obtain the correlation between the low- and high-fidelity data, (2) infer the quantities of interest based on a few scattered data, and (3) identify the unknown parameters in the PDEs. In particular, we employ the MPINNs to learn the hydraulic conductivity field for  unsaturated flows as well as the reactive models for reactive transport. The results demonstrate that MPINNs can achieve relatively high accuracy based on a very small set of high-fidelity data. Despite the relatively low dimension and limited number of fidelities (two-fidelity levels)  for the benchmark problems in the present study, the proposed model can be readily extended to very high-dimensional regression and classification problems involving multi-fidelity data.



multi-fidelity physics-informed neural networks adversarial data  porous media reactive transport

















§ INTRODUCTION



The  recent rapid developments in deep learning have also influenced the computational modeling of physical systems, e.g. in geosciences and  engineering  <cit.>. Generally, large numbers of high-fidelity data sets are required for optimization of complex physical systems, which may lead to computationally prohibitive costs. On the other hand, inadequate high-fidelity data result in inaccurate approximations and possibly erroneous designs. Multi-fidelity modeling has been shown to be both  efficient and effective in achieving high accuracy in diverse applications by leveraging both the low- and  high-fidelity data <cit.>. In the framework of multi-fidelity modeling, we assume that accurate but expensive high-fidelity data are scarce, while the cheaper and less accurate low-fidelity data are abundant. An example is the use of a few experimental measurements, which are hard to obtain, combined with synthetic data obtained from running a computational model. In many cases, the low-fidelity data can supply useful information on the trends for high-fidelity data, hence multi-fidelity modeling can greatly enhance prediction accuracy based on a small set of high-fidelity data in comparison to the single-fidelity modeling <cit.>.



The construction of  cross-correlation between the low- and high-fidelity data is crucial in multi-fidelity methods. Several methods have been developed to estimate such correlations, such as the response surface models <cit.>, polynomial chaos expansion <cit.>,  Gaussian process regression (GPR) <cit.>, artificial neural networks <cit.>,  and moving least squares <cit.>. Interested readers can refer to <cit.> for a comprehensive review of these methods. Among all the existing methods, the Gaussian process regression  in combination with the linear autoregressive scheme has drawn much attention in a wide range of applications <cit.>.   For instance, Babaee et al. applied this approach for the mixed convection to propose an improved correlation for heat transfer, which outperforms existing empirical correlation <cit.>.   We note that GPR with a linear autoregressive scheme can only capture the linear correlation between the low- and high-fidelity data. Perdikaris et al. then extended the method in <cit.> to enable it of learning complex nonlinear correlations <cit.>; this has been successfully employed to estimate the hydraulic conductivity based on the multi-fidelity data for pressure head in subsurface flows  <cit.>. Although great progress has already been made, the multi-fidelity approaches based on GPR still have some limitations, e.g., approximations of discontinuous functions <cit.>,   high-dimensional problems <cit.>, and  inverse problems with strong nonlinearities (i.e., nonlinear partial differential equations) <cit.>. In addition, optimization for GPR is  quite difficult to implement. Therefore, multi-fidelity approaches which can overcome these drawbacks are urgently needed.


Deep neural networks can easily handle  problems with almost any nonlinearities at both low- and high-dimensions. In addition, the recently proposed physics-informed neural networks (PINNs) have shown expressive power for learning the unknown parameters or functions in inverse PDE problems with  nonlinearities <cit.>. Examples of successful applications of PINNs include (1) learning the velocity and pressure fields based on partial observations of spatial-temporal visualizations of a passive scalar, i.e., solute concentration <cit.>,  and (2) estimation of the unknown constitutive relationship in the nonlinear diffusion equation for unsaturated flows <cit.>. Despite the expressive power of PINNs, it has been documented that a large set of high-fidelity data is required for identifying the unknown parameters in nonlinear PDEs. To leverage the merits of deep neural networks (DNNs) and the concept of multi-fidelity modeling, we propose to develop multi-fidelity DNNs and multi-fidelity PINNs (MPINNs), which are expected to have the following attractive features:
 (1) they can learn both the linear and nonlinear correlations adaptively; (2) they are suitable for high-dimensional problems; (3) they can handle inverse problems with strong nonlinearities; and (4) they are easy to implement, as we demonstrate in the present work.   


The rest of the paper is organized as follows: the key concepts of multi-fidelity DNNs and MPINNs are presented in Sec. <ref>, while results for function approximation and inverse PDE problems are shown in Sec. <ref>. Finally, a summary for this work is given in Sec. <ref>. In the Appendix we include a basic review of the embedding theory.



§ MULTI-FIDELITY DEEP NEURAL NETWORKS AND MPINNS


The key starting point in multi-fidelity modeling is to discover and exploit the relation between low- and high-fidelity data <cit.>. A widely used comprehensive correlation  <cit.> is expressed as 

    y_H = ρ(x) y_L + δ(x),

where y_L and y_H are, respectively, the low- and high-fidelity data, ρ(x) is the multiplicative correlation surrogate, and δ(x) is the additive correlation surrogate. It is clear that multi-fidelity models based on this relation are only capable of handling linear correlations between the two-fidelity data. However, there exist many interesting cases that go beyond the linear correlation in Eq. (<ref>) <cit.>. For instance, the correlation for the low-fidelity experimental data and the high-fidelity direct numerical simulations in the mixed convection flows past a cylinder is nonlinear <cit.>.   In order to capture the nonlinear correlation, we put forth a generalized autoregressive scheme, which is expressed as

    y_H = F(y_L) + δ(x),

where F(.) is an unknown (linear/nonlinear) function that maps the low-fidelity data to the high-fidelity level.  We can further write Eq. (<ref>)  as

    y_H = ℱ(x, y_L).

To explore the linear/nonlinear correlation adaptively, we then decompose ℱ(.)  into two parts, i.e., the linear and nonlinear parts, which are expressed as 

    ℱ = ℱ_l + ℱ_nl,
 
where ℱ_l and ℱ_nl  denote the linear and nonlinear terms in ℱ, respectively.  Now, we construct the correlation as

    y_H = αℱ_l(x, y_L) + (1 - α) ℱ_nl(x, y_L) ,  α∈ [0, 1],

where α is hyper-parameter to be determined by the data; its value determines the degree of
nonlinearity of the correlation between the high- and low-fidelity data.





The architecture of the proposed multi-fidelity DNN and MPINN is illustrated in Fig. <ref>, which is composed of four fully-connected neural networks. The first one 
𝒩𝒩_L(x_L, θ) is employed to approximate the low-fidelity data, while the second and third NNs (𝒩𝒩_H_i(x, y_L, β, γ_i), i = 1, 2) are for approximating the correlation for the low- and high-fidelity data; the last NN (𝒩𝒩_f_e) is induced by encoding the governing equations, e.g. the partial differential equations (PDEs). In addition, ℱ_l = 𝒩𝒩_H_1, and ℱ_nl = 𝒩𝒩_H_2; θ, α, β, and γ_i, i = 1, 2 are unknown parameters of the NNs, which can be learned by minimizing the following loss function:

    MSE = MSE_y_L + MSE_y_H + MSE_f_e + λ∑β_i^2,

where

    MSE_y_L   = 1/N_y_L∑^N_y_L_i = 1( |y^*_L - y_L|^2  + |∇ y^*_L - ∇ y_L|^2 ), 
    
    MSE_y_H   = 1/N_y_H∑^N_y_H_i = 1( |y^*_H - y_H|^2 ),
    
    MSE_f_e   = 1/N_f∑^N_f_i = 1( |f^*_e - f_e|^2 ).

Here, ψ (ψ = y^*_L,  y^*_H, and f^*_e) denote the outputs of the 𝒩𝒩_L, 𝒩𝒩_H, and 𝒩𝒩_f_e, β is any weight in 𝒩𝒩_L and 𝒩𝒩_H_2 as well as α, and λ is the L_2 regularization rate. It is worth mentioning that the boundary/initial conditions  for f_e can also be added into the loss function, in a similar fashion as in the standard PINNs introduced in detail in <cit.> so we do not elaborate on this issue here. In the present study, the loss function is optimized using the L-BFGS method together with Xavier's initialization method, while the  hyperbolic tangent function is employed as the activation function in 𝒩𝒩_L and 𝒩𝒩_H_2.  We note that no activation function is included in 𝒩𝒩_H_1 due to the fact that it is used to approximate the linear part of ℱ. Hence, the present DNNs can dynamically learn the linear as well as the nonlinear correlations without any prior knowledge on the correlation for the low- and high-fidelity data. 



§ RESULTS AND DISCUSSION


Next we present several tests of the multi-fidelity DNN as well as the MPINN, the latter in the context of two inverse PDE problems related to geophysical applications. 



 §.§ Function approximation

We first demonstrate the effectiveness of this multi-fidelity modeling  in approximating both continuous and discontinuous functions based on both linear and complicated nonlinear correlations between the low- and high-fidelity data.




  §.§.§ Continuous function with linear correlation

We first consider a pedagogical example of  approximating  an one-dimensional function based on data from two levels of fidelities.  The low-  and high-fidelity  data are generated from:


    y_L(x)     = A(6x - 2)^2 sin(12x - 4) + B(x - 0.5) + C,   x ∈ [0, 1]
    
    y_H(x)     = (6x - 2)^2 sin(12x - 4),

where y_H is linear with y_L, and A = 0.5, B = 10, C = -5. As shown in Fig. <ref>, the  training data at the  low- and high-fidelity level are  x_L = { 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 
 0.7, 0.8, 0.9, 1} and x_H = { 0, 0.4, 0.6, 1}, respectively.  




We first try to predict the true function using the high-fidelity data only. For this case, we only need to keep 𝒩𝒩_H_2  (Fig. <ref>). In addition, the input for 𝒩𝒩_H_2 becomes x because no low-fidelity data are available.  Here 4 hidden layers and 20 neurons per layer are adopted in  𝒩𝒩_H_2 and no regularization is used. The learning rate is set as 0.001. As we can see in Fig. <ref>, the present model provides inaccurate predictions due to the lack of sufficient high-fidelity data. Furthermore, we also plot the predictive posterior means of the Kriging <cit.>, which is noted to be similar as the results from the 𝒩𝒩_H_2. 

Keeping the high-fidelity data fixed, we try to improve the accuracy of prediction by adding low-fidelity data (Fig. <ref>).  In this case, the last DNN for the PDE is discarded.  Here 2 hidden layers and 20 neurons per layer are used in 𝒩𝒩_L, while 2 hidden layers with 10 neurons per layer are employed for 𝒩𝒩_H_2, and only 1 hidden with 10 neurons are used in 𝒩𝒩_H_1 (The size of 𝒩𝒩_H_1 is kept identical in all of the following cases).  The regularization rate is set to λ = 10^-2 with a learning rate 0.001. As shown in Fig. <ref>, the present model provides accurate predictions for the high-fidelity profile.  In addition, the prediction using the Co-Kriging is displayed in Fig. <ref> <cit.>. We see that the learned profiles from these two methods are similar, while the result from the present model is slightly better than the Co-Kriging, which can be seen in the inset of Fig. <ref>. Finally, the estimated  correlation is illustrated in Fig. <ref>, which also agrees quite well with the exact result.  Unlike the Co-Kriging/GPR, no prior knowledge on the correlation between the low- and high-fidelity data is needed in the multi-fidelity DNN, indicating that the present model can learn the correlation dynamically based on the given data.



The size of the neural network (e.g., depth and width) has a strong effect on the predictive accuracy <cit.>, which is also investigated here. Since we have sufficient  low-fidelity data, it is easy to find an appropriate size for 𝒩𝒩_L to approximate the low-fidelity function. Therefore, the particular focus is put on the size of  𝒩𝒩_H_2 due to the fact that the few high-fidelity data  may yield overfitting.  Note that since the correlation between the low- and high-fidelity data is relatively simple, there is no need to set the 𝒩𝒩_H_2 to have a large size. Hence, we limit the ranges of the depth (i.e., l) and width (i.e., w) as: l ∈ [1, 4] and w ∈ [2, 32], respectively. Considering that a random initialization is utilized, we perform ten runs for each case with different depth and width. The mean and standard deviation for the relative L_2 errors defined as

    E = 1/N∑^n = N_n=1√(∑_j (y^*_j - y_j)^2/∑ y^2_j),  σ =√(∑^n = N_n=1(E_n - E)^2/N),

are used to quantify the effect of the size of the 𝒩𝒩_H_2.  In Eq. (<ref>), E is the mean relative L_2 errors, n is the index of each run, N is the total number of runs (N = 10), j is the index for each sample data points, E_n is the relative L_2 error for the n-th run, and the definitions of y^* and y are the same as those in Sec. <ref>. As shown in Table <ref>, the computational errors for 𝒩𝒩_H_2 with different depth and width are almost the same. In addition, the standard deviation for the relative L_2 errors are not presented because they are less than 10^-5 for each case.  
All these results demonstrate the robustness of the 
multi-fidelity DNNs.  To reduce the computational cost as well and retain the accuracy, a good choice for the size of 𝒩𝒩_H_2  may be l ∈ [1,2] and w ∈ [4, 20] in low dimensions.








  §.§.§ Discontinuous function with linear correlation

As mentioned in <cit.>, the approximation of a discontinuous function using GPR is challenging due to the continuous kernel employed.  We then proceed to test the  capability of the present model for approximating discontinuous functions. The low- and high-fidelity data are generated by the following “Forrester" functions with jump <cit.>: 

    y_L(x) ={   0.5(6x - 2)^2sin(12x - 4) + 10(x - 0.5) - 5,   0 ≤ x ≤ 0.5,
       3 + 0.5(6x - 2)^2sin(12x - 4) + 10(x - 0.5) - 5,   0.5 < x ≤ 1,
    .

and

    y_H(x) ={   2y_L(x) - 20x + 20,   0 ≤ x ≤ 0.5,
       4 + 2y_L(x) - 20x + 20,   0.5 < x ≤ 1.
    .






As illustrated in Fig. <ref>,  38 and 5 sampling data points are employed as the training data at the low- and high-fidelity level, respectively. The learning rate is again set as 0.001 for all test cases here. Similarly, we employ the 𝒩𝒩_H_2 (l × w = 4 × 20) to predict the high-fidelity values on the basis of the given high-fidelity data only, but
the corresponding prediction is not good (Fig. <ref>). However, using the multi-fidelity data, the present model can provide quite accurate predictions for the high-fidelity profile (Fig. <ref>).  Remarkably, the multi-fidelity DNN can capture the discontinuity at x = 0.5 at the high-fidelity level quite well even though no data are available in the range 0.4 < x < 0.6. This is reasonable because the low- and high-fidelity data share the same trend as 0.4< x < 0.6, yielding the correct predictions of the high-fidelity values in this zone.  Furthermore, the learned correlation is displayed in Fig.  <ref>, which shows only slight differences from the exact correlation. 





  §.§.§ Continuous function with nonlinear correlation



To test the present model for capturing complicated nonlinear correlations between the low- and high-fidelity data, we further consider the following case <cit.>:

    y_L(x)    = sin(8 π x),  x ∈ [0, 1], 
    
    y_H(x)    = (x - √(2)) y^2_L(x).

Here, we employ 51 and 14 data points (uniformly distributed) for low- and high-fidelity, respectively,  as the training data, (Fig. <ref>). The learning rate for all test cases is still 0.001.

As before, the 𝒩𝒩_H_2 (l × w = 4 × 20) cannot provide accurate predictions for the high-fidelity values using only the few high-fidelity data points as displayed in Fig. <ref>. We then test the performance of the multi-fidelity DNN based on the multi-fidelity training data. Four hidden layers and 20 neurons per layer are used in 𝒩𝒩_L, and 2 hidden layers with 10 neurons per layer are utilized for 𝒩𝒩_H_2.  Again, the predicted profile from the present model agrees well with the exact profile at the  high-fidelity level, as shown in Fig. <ref>. It is interesting to find that the multi-fidelity DNN can still provide accurate predictions for the high-fidelity profile even though the trend for the low-fidelity data is opposite to that of the high-fidelity data, e.g., 0 < x < 0.2, a case of adversarial type of data. In addition, the learned correlation between the low- and high-fidelity data agree well with the exact one as illustrated in Fig. <ref>, indicating that the multi-fidelity DNN is capable of  discovering the non-trivial underlying correlation on the basis of training data.







  §.§.§ Phase-shifted oscillations

For more complicated correlations between the low and high-fidelity data, we can easily extend the 
multi-fidelity DNN based on the “embedding theory" to enhance the capability for learning more complex correlations <cit.> (For more details on the embedding theory, refer to <ref>).  Here, we consider the following low-/high-fidelity functions with phase errors <cit.>:

    y_H(x)    = x^2 + sin^2(8 π x + π/10), 
    
    y_L(x)     = sin(8 π x).

We can further write y_H in terms of y_L as

    y_H = x^2 + (y_L cos(π/10) + y^(1)_L sin(π/10))^2,

where y^(1)_L denotes the first derivatives of y_L.  The relation between the low- and high-fidelity data is displayed in Fig. <ref>, which is rather complicated. The performance of the 
multi-fidelity DNN for this case will be tested next.

To approximate the high-fidelity function, we select 51 and 16 uniformly distributed sample points as the training data for low- and high-fidelity values, respectively (Fig. <ref>). The selected learning rate for all test cases is 0.001. Here, we test two types of inputs for 𝒩𝒩_H_2, i.e., [x, y_L(x)] (Method I), and [x, y_L(x), y_L(x - τ)] (Method II) (τ is the delay).  Four hidden layers and 20 neurons per layer are used in 𝒩𝒩_L, and 2 hidden layers with 10 neurons per layer are utilized for 𝒩𝒩_H_2.  As shown in Fig. <ref>, it is interesting to find that Method II provides accurate predictions for the high-fidelity values (Fig. <ref>), while  Method I fails (Fig. <ref>).  As mentioned in <cit.>, the term y_L(x - τ) can be viewed as an implicit approximation for y^(1)_L, which enables Method II to capture the correlation in Eq. (<ref>) based only on a small number of high-fidelity data points. However, given that no information on y^(1)_L is available in Method I, the present datasets are insufficient to obtain the correct correlation. 










  §.§.§ 20-dimensional function approximation

In principle, the new multi-fidelity DNN can approximate any high-dimensional function so here we 
take a modest size so that is not computationally expensive to train the DNN. Specifically, we generate 
the low- and high-fidelity data for a 20-dimensional function from the following equations: <cit.>

    y_H(x)    = (x_1 - 1)^2 + ∑^20_i=2(2x^2_i - x_i-1)^2, x_i ∈[-3, 3], i = 1, 2, ..., 20, 
    
        y_L(x)    = 0.8y_H(x) - ∑^19_i=1 0.4 x_i x_i+1 - 50.


As shown in Fig. <ref>, using only the available high-fidelity data does not lead to an accurate
function approximation but using the multi-fidelity DNN approach gives excellent results as shown in 
Fig. <ref>.








In summary, in this section we have demonstrated using different data sets and correlations that multi-fidelity DNNs can adaptively learn the underlying correlation between the low- and high-fidelity data from the given datasets without any prior assumption on the correlation. In addition, they can be applied to high-dimensional cases, hence outperforming GPR <cit.>. Finally, the present framework can be easily extended based on the embedding theory to non-functional correlations, which enables multi-fidelity DNNs to learn more complicated nonlinear correlations induced by phase errors of the low-fidelity data (adversarial data). 





 §.§ Inverse PDE problems with nonlinearities

In this section, we will apply the multi-fidelity PINNs (MPINNs) to two inverse PDE problems with nonlinearities, specifically, unsaturated flows and reactive transport in porous media. 
We assume that the hydraulic conductivity is first estimated based on scarce high-fidelity measurements of the pressure head. Subsequently, the reactive models are further learned given a small set of  high-fidelity observations of the solute concentration.



  §.§.§ Learning the hydraulic conductivity for nonlinear unsaturated flows









Unsaturated flows play an important role in the ground-subsurface water interaction zone <cit.>.  Here we consider a steady unsaturated flow in an one-dimensional (1D) column with a variable water content,  which can be described by the following equation as

    ∂_x ( K(h) ∂_x h ) = 0.

We consider two types of boundary conditions, i.e., (1)  constant flux at the inlet and constant pressure head at the outlet, q = -K∂_x h = q_0,  x = 0;  h = h_1,  x = L_x (Case I),  and (2) constant pressure head at both the inlet and outlet, h = h_0,  x = 0;  h = h_1,  x = L_x (Case II). Here L_x = 200cm is the length of the column, h is the pressure head, h_0 and h_1 are, respectively, the pressure head at the inlet and outlet, q represents the flux, and q_0 is the flux at the inlet, which is a constant. In addition, K(h) denotes the pressure-dependent hydraulic conductivity, which is expressed as  

    K(h) = K_s S^1/2_e [ 1 - (1 - S^1/m_e)^m]^2,

where K_s is the saturated hydraulic conductivity, and S_e is the effective saturation that is a function of h. It is noted that several models have been developed to characterize S_e but among them, the van Genuchten model is the most widely used <cit.>, which reads as follows:

    S_e = 1/(1 + |α_0 h|^n)^m,    m = 1 - 1/n.

In Eq. (<ref>), α_0 is related to the inverse of the air entry suction, and m represents a measure of the pore-size distribution. To obtain the velocity field for later applications, we should first obtain the distribution of K(h). Unfortunately, both parameters depend on the geometry of porous medium and are difficult to measure directly. We note that the pressure head can be measured more easily in comparison to α_0 and m. Therefore, we assume that partial measurements of h are available without the direct measurements of α_0 and m. The objective is to estimate α_0 and m based on the observations of h. Then, we can compute the distribution of K(h) according to Eqs. (<ref>) and (<ref>).









The loam  is selected as a representative case here, for which the empirical ranges of α_0 and m are: α_0 (cm^-1) ∈ [0.015, 0.057] and m ∈ [0.31, 0.40] <cit.>. In addition, K_s = 1.04 cm/hr. To obtain the training data for neural networks, two types of numerical simulations are conducted to generate the low- and high-fidelity data using the bvp4c in Matlab (uniform lattice with δ_x = 1/15 cm).  For high-fidelity data,  the exact values for α_0 and m are assumed to be 0.036 cm^-1 and 0.36. The high-fidelity simulations are then conducted using the exact values of α_0 and m. Different initial guesses for α_0 and m are employed in the low-fidelity simulations. Specifically, ten uniformly distributed pairs i.e., (α_0, m) in the range (0.015, 0.31) - (0.057, 0.40)  are adopted in the low-fidelity simulations. For all cases, 31 uniformly distributed sampling data at the low-fidelity level are served as the training data, and only two sampling points including the boundary conditions mentioned above are employed as the training data for high-fidelity. In addition,  a smaller learning rate i.e., 10^-4 is employed for all test cases in this section.





We first consider the flow with constant flux inlet. The flux at the inlet and the pressure at the outlet are set as q_0 = 0.01cm/y and h_1 = -20 cm, respectively. Equation (<ref>) is added into the last neural network in MPINNs. We employ the numerical results for α_0 = 0.055 and m = 0.4 as the low-fidelity data. As shown in Fig. <ref>, the prediction for hydraulic conductivity is different from the exact solution. According to Darcy's law, we can rewrite Eq. (<ref>) as

    q(x) = -K∂_x h, ∂_x q(x) = 0.

 Considering that q = q_0 at the inlet is a constant, we can then obtain the following equation
 
    q(x) = -K∂_x h = q_0,

which actually is the mass conservation at each cross section. We then employ Eq. (<ref>) instead of Eq. (<ref>) in the MPINNs, and the results improve greatly (Fig. <ref>). 



We proceed to study this case in some more detail. We perform the single-fidelity modeling (SF) based on the high-fidelity data. We use two hidden layers with 20 neurons per layer in 𝒩𝒩_H_2, in which the hyperbolic tangent function is employed as the activation function. The learned pressure head and the hydraulic conductivity are shown in Figs. <ref>-<ref>. We observe that both the learned h and K(h) disagree with the exact results.  We then switch to  multi-fidelity modeling. Two hidden layers and 10 neurons per layer are used in 𝒩𝒩_L, and two hidden layers with 10 neurons per layer are utilized for 𝒩𝒩_H_2.  The predicted pressure head as well as the hydraulic conductivity (average value from ten runs with different initial guesses) agree quite well with the exact values (Figs. <ref>-<ref>).

For Case II, we set the pressure head at the inlet and outlet as h_0 = -3 cm and h_1 = -10 cm. We also assume that the flux at the inlet is known, thus Eq. (<ref>) can  also be employed instead of Eq. (<ref>) in the MPINNs. The training data are illustrated in Fig. <ref>. The size of the NNs here is kept the same as that used in Case I. We observe that results for the present case (Figs. <ref>-<ref>) are quite similar with those in Case I.


Finally, the mean values of α_0 as well as the m for different initial guesses are shown in Table <ref>, which indicates that the MPINNs can significantly improve the prediction accuracy as compared to the estimations based on the high-fidelity only (SF in Table <ref>).   







  §.§.§ Estimation of reaction models for reactive transport

We further consider a single irreversible chemical reaction in a 1D soil column with a  length of 5m, which is expressed as 

    a_r A → B,

where A, and B are different solute. The above reactive transport can be described by the following advection-dispersion-reaction equation as

    ∂_t (ψ C_i) + q  ∂_x C_i =  ψ D∂^2_x C_i - ψ v_i k_f,r C_A^a_r, (i = A, B),

where C_i (mol/L) is the concentration of any solute, q is the Darcy velocity, ψ is the porosity,  D is the dispersion coefficient, k_f,r denotes the chemical reaction rate, a_r is the order of the chemical reaction, both of which are difficult to measure directly, and v_i is the stoichiometric coefficient with v_A = a_r, and v_B = -1.  Here, we assume that the following parameters are known:  ψ = 0.4, q = 0.5m/y,  and D = 10^-8 m/s^2. The initial and boundary conditions imposed on the solute are expressed as

    C_A(x, 0) = C_B(x, 0)  = 0,
    
    C_A(0, t)  = 1,  C_B(0, t) = 0,
    ∂_x C_i(x, t)|_x =l_x = 0.
 
The objective here is to learn the effective chemical reaction rate as well as the reaction order based on partial observations of the concentration field C_A(x, t).


We perform lattice Boltzmann simulations <cit.> to obtain the training data since we have no experimental data. Consider that v_A is a constant, we define an effective reaction rate as k_f = v_A k_f,r for simplicity.  The exact effective reaction rate and reaction order are assumed to be k_f = 1.577 / y and a_r = 2, respectively. Numerical simulations  with the exact k_f and a_r are then conducted to obtain the high-fidelity data. In simulations, a uniform lattice is employed, i.e., l_x = 400 δ_x, where δ_x = 0.0125m is the space step, and δ_t = 6.67 × 10^-4 y is the time step size. We assume that the sensors for concentration are located at x = {0.625, 1.25, 2.5, 3.75}m. In addition, we assume that the data are collected from the sensors once half a year. In particular, we employ two different datasets (Fig. <ref>), i.e., (1) t = 0.5 and 1 years (Case I), and (2) t = 0.25 and 0.75 years.
Schematics of the training data points for the two cases we consider are shown in Fig. <ref>
and Fig. <ref>.





Next, we describe how we obtain the low-fidelity data. In realistic applications, the pure chemical reaction rate (without porous media)  between different solute e.g., A and B are known, which can be served as the initial guess for k_f. Here we assume that the initial guess for the chemical reaction rate and reaction order vary from 0.75 k_f/a_r to 1.25 k_f/a_r. To study the effect of the initial guess ((k_f,0, a_r0)) on the predictions, we conduct the low-fidelity numerical simulations based on ten uniformly distributed pairs in [0.75 k_f, 0.75a_r] - [1.25k_f, 1.25a_r] using the same grid size and time step as the high-fidelity simulations. Here k_f,0 and a_r0 represent the initial guesses for k_f and a_r. The learning rate employed in this section is also 10^-4.













The results of predictions using PINNs (with the hyperbolic tangent activation function) trained on high-fidelity data are shown in Fig. 
<ref> and Fig. <ref> for the two cases we consider, and corresponding results 
using MPINNs are shown in Fig. <ref> and Fig. <ref>.

The estimated mean and standard deviation for k_f and a_r are displayed in Table <ref>, which are much better than the results from single-fidelity modelings. We  also note that the standard deviations are rather small, which demonstrates the robustness of the MPINNs. 






§ CONCLUSION


In this work we presented a new composite deep neural network that learns from multi-fidelity data,
i.e. a small set of high-fidelity data and a larger set of inexpensive low-fidelity data. This scenario
is prevalent in many cases for modeling physical and biological systems and we expect that the
new DNN will provide solutions to many current bottlenecks where availability of large data sets of high-fidelity is simply not possible but either low-fidelity data from inexpensive sensors or other modalities or even simulated data can be readily obtained. Moreover, we extended the concept of
physics-informed neural networks (PINNs) that use a single-fidelity data to train to the multi-fidelity case and MPINNs. Specifically,  MPINNs are composed of four fully connected neural networks: the fist neural network approximates the low-fidelity data, while the second and third NNs are for constructing the correlations between the low- and high-fidelity data, and the last NN  encodes the PDEs that describe the corresponding physical problems. The two sub-networks included in the high-fidelity network are employed to approximate the linear and nonlinear parts of the correlations, respectively. Training the relaxation parameter employed to combine the two sub-networks enables the MPINNs to learn the correlation based on the training data without any prior assumption on the relation between the low- and high-fidelity data. 

MPINNs have the following attractive features: (1) Owing to the expressible capability of function approximation of the NNs, multi-fidelity NNs are able to approximate both continuous and discontinuous functions in high dimensions; (2) Due to the fact that NNs can handle almost any kind of nonlinearities, MPINNs are effective for identification of unknown parameters or functions in inverse problems described by nonlinear PDEs. 


We first tested the new multi-fidelity DNN in approximating continuous and discontinuous functions with linear and nonlinear correlations.  Our results demonstrated that the present model can adaptively learn the correlations between the low- and high-fidelity data based on the training data
of variable fidelity. In addition, this model can easily be extended based on the embedding theory to learn more complicated nonlinear and non-functional correlations. We then tested MPINNs on inverse PDE problems, namely,  in estimating the hydraulic conductivity for unsaturated flows as well as the reaction models in reactive transport in porous media. We found that the proposed new MPINN can identify the unknown parameters or even functions with high accuracy  using very few high-fidelity data, which is  promising in reducing the high experimental cost for collecting high-fidelity data. Finally, we point out that MPINNs can also be employed for high-dimensional problems as well as problems with multiple fidelities, i.e. more than two fidelities.  




§ ACKNOWLEDGEMENT

This work was supported by the PHILMS grant DE-SC0019453, the DOE-BER grant DE-SC0019434, the AFOSR grant FA9550-17-1-0013, and the DARPA-AIRA grant HR00111990025. In addition, we would like to thank Dr. Guofei Pang, Dr. Zhen Li, Dr. Zhiping Mao, and Ms Xiaoli Chen for their helpful discussions.





§ DATA-DRIVEN MANIFOLD EMBEDDINGS


To learn more complicated non-linear correlation between the low- and high-fidelity data, we can further link the multi-fidelity DNNs with the embedding theory <cit.>. According to the weak Whitney embedding theorem <cit.>, any continuous function from an n-dimensional manifold to an m-dimensional manifold may be approximated by a smooth embedding with m > 2n.  Using this theorem, Taken's theorem <cit.> further points out that the m embedding dimensions can be composed of m different observations of the system state variables or m time delays for a single scalar observable. 



Now we will introduce the applications of the two theorems in multi-fidelity modelings. We assume that both y_L and y_H are smooth functions. Suppose that y_L, y_L(x - τ), ..., y_L(x - (m-1)τ) (τ is the time delay)  and a small number of (x, y_H) are available, we can then express y_H in the following form 

    y_H(x) = ℱ(x, y_L(x), y_L(x - i τ)), i = 1,...,m-1.

By using this formulation, we can construct more complicate correlation than Eq. (<ref>). To link the multi-fidelity DNN with the embedding theory, we can extend the inputs for 𝒩𝒩_H,i to higher dimensions, i.e., [ x, y_L(x) ] → [ x, y_L(x), y_L(x - τ), y_L(x - 2τ), ..., y_L(x - (m-1) τ) ], which enables the multi-fidelity DNN to discover more complicated underlying correlations between the low- and high-fidelity data. 



Note that the selection of optimal value for the time delay τ is important in embedding theory <cit.>, on which numerous studies have been carried out <cit.>. However, most of the existing methods for determining the optimal value of τ appear to be problem-dependent <cit.>. Recently, Dhir et al. proposed a Bayesian delay embedding method, where τ is robustly learned from the training data by employing the variational autoencoder <cit.>.  In the present study, the value of τ is also learned by optimizing the NNs rather than setting it as constant as in the original work presented in Ref. <cit.>.




















elsarticle-num
sort   compress



















