title: Explore missing flow dynamics by physics-informed deep learning: the parameterised governing systems
authors: Hui Xu Wei Zhang Yong Wang
text: 
  
  
  
  
  
  School of Aeronautics and Astronautics, Shanghai Jiao Tong University, Shanghai, 200240, China
  
  
  waynezw0618@163.com
  Science and Technology on Water Jet Propulsion Laboratory, Marine and Research Institute of China, Shanghai, China
  
  
  
  yong.wang@ds.mpg.de
  Max Planck Institute for Dynamics and Self-Organization, 37077 Göttingen, Germany
  
  
  
  
  
  
  
  Gaining and understanding the flow dynamics have much importance in a wide range of disciplines, e.g. astrophysics, geophysics, biology, mechanical engineering and biomedical engineering. As a reliable way in practice, especially for turbulent flows, regional flow information such as velocity and its statistics, can be measured experimentally. Due to the poor fidelity or experimental limitations, some information may not be resolved in a region of interest. On the other hand, detailed flow features are described by the governing equations, e.g. the Navier-Stokes equations for viscous fluid, and can be resolved numerically, which is heavily dependent on the capability of either computing resources or modelling. Alternatively, we address this problem by employing the physics-informed deep learning, and treat the governing equations as a parameterised constraint to recover the missing flow dynamics. We demonstrate that with limited data, no matter from experiment or others, the flow dynamics in the region where the required data is missing or not measured, can be reconstructed with the parameterised governing equations. Meanwhile, a richer dataset, with spatial distribution of the control parameter (e.g. eddy viscosity of turbulence modellings), can be obtained. The method provided in this paper may shed light on data-driven scale-adaptive turbulent structure recovering and understanding of complex fluid physics, and can be extended to other parameterised governing systems beyond fluid mechanics.	
  
  Explore missing flow dynamics by physics-informed deep learning: the parameterised governing systems
      Yong Wang
      
  ====================================================================================================
  
  
  
  
  
  § INTRODUCTION
  
  In fluid mechanics, acquiring the flow data is one of the key tasks in science and engineering <cit.>. Analysing or controlling the flow dynamics generally needs a rich data set or a continuous representation of the flow field. Conventionally, the flow dynamics is often conducted by either numerics or experiments. Numerically, the Navier-Stokes equations (NSEs), which govern the flow dynamics in a form of partial differential equations (PDEs), can be solved under typical assumptions or specific conditions within a typical range of precision. For high Reynolds number flows, or turbulent flows, directly solving the NSEs is extremely challenging due to limited computer resources and  large amounts of degrees of freedom <cit.>. To annihilate the restriction, various turbulence models, which are coupled with the NSEs, are proposed to save computational cost <cit.>. Meanwhile, suspicion around correctness and applicability of modelling arises, which leads to unremitting pursuit of researches in turbulence modelling <cit.>, although machine learning approaches have been used to enhance the predictive capability <cit.>.  
  
  
  In contrast, nowadays, gain of data from experiments is reliable, which generally has no restriction from the Reynolds number. However, due to the precision and uncertainty of experimental equipment  <cit.>, measured data can be obtained in a finite set of locations, and are precise only in a limited-order statistical sense, which are not as sufficiently accurate as the solutions described by the NSEs. In a statistical sense, the experimental data can be generally regarded as an approximation of the solutions of the NSEs. It turns out that leveraging the limited experimental data to the underlying governing equations appears as a critical question in modelling and recovering the flow dynamics. In order to address this problem, we here consider the reformed governing equations where the control parameter is regarded as an unknown, and employ the physics-informed learning technique to preserve fundamental physical principles <cit.>, by which the parameterised governing equations and the missing flow dynamics can be physically determined at the same time. To achieve the above, the parameterised governing equations are treated as a physical constraint, which is used to form a loss function in the training of neural networks <cit.>.
  
  A data-driven discovery of PDEs, named PDE-FIND, was proposed by Rudy and his collaborators. Especially, for the NSEs, the PDE-FIND was used to identify the Reynolds number in the vorticity transport equation for fluid mechanics<cit.>. The research group of Karniadakis has made a great contribution in physics-informed neural networks <cit.> and their applications to fluid flows. Generally, physics-informed neural networks can be used for both forward and inverse problems. For forward problem, Raissi et al. proposed the hidden fluid mechanics <cit.>, which provides a new way to learn velocity and pressure fields from flow visualizations. Most recently Jin et al. proposed the NSFNets to solve two different, velocity-pressure and vorticity-velocity, formulations of the incompressible NSEs, in the sense of direct numerical simulation for Kovasznay flow and turbulent channel flow <cit.>. Other applications are for high speed inviscid flow <cit.> and vortex-induced vibration <cit.>. For inverse problems, physics-informed neural networks were used to identify the Reynolds number from flow data for the low Reynolds number flow past a cylinder <cit.>, and to discover universal variable-order fractional model for turbulent Couette flow <cit.>. 
  
  Built on the above evidence, the generalisation of physics-informed neural networks forms the basis of the parameterised governing system in exploring missing flow physics and exploiting the essential of modelling parameters, specially for turbulent flows. To overcome numerical stiffness, a recently developed learning rate annealing algorithm <cit.> is adopted in this work. The proposed idea can be extended to a broad range of applications where PDEs appear as physical constraints and the control parameter of the PDEs plays a role in dominating the physics.
  
  
  
  
  § METHODOLOGY
  
  
  Mathematically, we consider the following system as a general description of physical phenomena in a region Ω:
  
      ℒ^t(p⃗; p⃗_0, g⃗_Γ, ν)=f⃗,
  
  where the operator ℒ^t represents the evolution of p⃗, and p⃗_0 is the initial condition. g⃗_Γ is the boundary conditions where Γ denotes the boundary ∂Ω of Ω. ν denotes the control parameter. For a steady system, the corresponding representation can be written as ℒ^∘(p⃗; g⃗_Γ, ν)=f⃗. In real-world applications, in the sense of temporal (or ensemble) averaging or spatial filtering, the control parameter can be represented as an effective one, ν_ eff, which is the summation of the physical ν and the modelled ν_t, while the original expression of the governing system is kept the same.  For instance,  in turbulence modelling, at any level of turbulence, closure of the averaged or filtered NSEs requires the parameter, eddy(subgrid) viscosity, which is introduced in large-eddy simulations (LES), scale-resolving simulations (SRS) and Reynolds-averaged Navier-Stokes (RANS) methods <cit.>. Once the effective control parameter ν_ eff is introduced as an unknown parameter, ℒ^t(p⃗; p⃗_0, g⃗_Γ, ν_ eff)=f⃗ or ℒ^∘(p⃗; g⃗_Γ, ν_ eff)=f⃗ can be regarded as the parameterised physical constraint which is used to explore missing flow dynamics under the framework of the physics-informed neural networks.
  
  Let us consider the Leonardo's self-portrait, shown in Figure <ref> (A), as a physical field. It is assumed that a part of Leonardo's hair is missing. In order to fix the portrait,  we need to find a pattern which is consistent with the missing one. Without referring the  potential governing system of the Leonardo's portrait, by learning the hair pattern beyond the information-missing region, it is possible to create a pattern which matches with the pattern around the region. It is clear that in principle, there should exist infinite possibilities. So, by virtue of limited information, how to physically make it unique as possible as we can? This is termed as “the parameterised physics-informed deep learning strategy”.
  
  
  
  
      < g r a p h i c s >
   
  
      < g r a p h i c s >
   
  
   (A)  Leonardo's hair   (B) Domain Ω and sub-domain Ω^' 
  figureIllustration of missing information: (A) Leonardo's self-portrait with a part of curly hair missing; [Generated from <https://en.wikipedia.org/wiki/Leonardo_da_Vinci>] (B) Mathematical representation of the domain Ω and the sub-domain Ω^' on which the information is missing.
  
  
  
  
  As illustrated in Fig. <ref>(B), p⃗ is known on Ω∖Ω^' and unknown on Ω^'. Theoretically, in fluid mechanics, if the effective eddy viscosity ν_ eff can be estimated or physically formulated,  the missing p⃗ on Ω^' can be solved from Eq. (<ref>). In the framework of the parameterised physics-informed learning, solving  Eq. (<ref>) is avoided and Eq. (<ref>) only leads us to directly find a physical unique solution on Ω^' in terms of p⃗ on Ω∖Ω^'. Let us turn back to the physical world of the fluid dynamics which is governed by the PDEs, e.g. the incompressible NSEs as follows
  
      ℒ^t({𝐮,p};𝐠,ν)≡{[ ∂𝐮/∂ t-ν∇^2𝐮+𝐮·∇𝐮+∇ p                 =𝐟, Ω;                   ∇·𝐮                   =0,;               𝐮|_Γ=∂Ω                   =𝐠. ].
  
  Theoretically, Eq. (<ref>) can be used for all kinds of incompressible flows. However, with extremely small viscosity ν, and thus very high Reynolds number, solving Eq. (<ref>) is extra challenging in practical applications. In another aspect, although some data can be obtained experimentally, it is also challenging, if possible, to establish the direct connection between Eq. (<ref>) and the gained experimental data. In engineering applications, the RANS equations, which are derived by the Reynolds decomposition of flow fields, have been well validated for engineering applications by experimental data and direct simulations of the NSEs <cit.>. The key question in RANS is to model the Reynolds stress τ_ij, which is represented as follows
  
      τ_ij=-ρu_i^' u_j^',
  
  where (·) denotes the Reynolds average or the spatial filtering, and u^'_i=u_i-u_i. Similar to Fick's law,  ρu_i^' u_j^' can be modelled as
  
      ρu_i^' u_j^'=-ρν_t (∂u̅_i/∂ x_j + ∂u̅_j/∂ x_i),
  
  where ν_t denotes the eddy viscosity. Although it is named as viscosity, ν_t is an artificial parameter different from the molecular viscosity in the original equations. Similar idea has been adopted to the large eddy simulation (LES), where sub-grid eddy viscosity ν_ sgs is introduced to model the subgrid scale turbulent flow. For hybrid RANS/LES such as detach eddy simulation (DES) and scale adaptive simulation (SAS), ν_t in the control system has different meaning in different region of the computational domain, depending on the resolution of the turbulence scales. Normally, ν_t is flow dependent. In the turbulence community, various modeling of the parameter ν_t
  has been studied for several decades. With the big data and machine learning, model constants of the RANS equations are tuned by fitting <cit.>. However, compared with small data sets, observed discrepancies are not explained any better with big ones <cit.>, and modelling ν_t so as the ν_ sgs in a universal sense still remains as a persistent challenging because of the case-by-case dependence. Luckily, engineering practices demonstrate that predictions of the RANS equations for complex flows are mostly quite satisfactory. It indicates that utilization of ν_t as an extra parameter can be a reliable way to establish the fitting between averaged/filtered experiment data and solutions of the corresponding control equations.  
  
  As indicated in the pioneering work <cit.>, representation of turbulent transport by an eddy viscosity underlies most of the related theoretical work. With this idea, the classical RANS equations can be rewritten as (without considering the auxiliary equations)
  
      ℒ^t({𝐮̅,p};g,ν+ν_η)≡{[ ∂𝐮̅/∂ t-(ν+ν_η)∇^2𝐮̅+𝐮̅·∇𝐮̅+∇p̅                           =𝐟, Ω;                            ∇·𝐮̅                             =0,;                        𝐮̅|_Γ=∂Ω                             =𝐠, ].
  
  
  where ν_η is an artificial viscosity term introduced here for parameterisation similar to that described above. From the control parameter point of view, the data-based determination of ν_η indicates  “scale adaption” of recovering turbulent structures, same as ν_ t in DES or SAS. The above equation is the system of the operator equation, Eq. (<ref>), in a concrete way. For steady cases, the unsteady term ∂𝐮̅/∂ t is neglected.  It should be mentioned that reformulating the NSEs does not imply employment of the RANS approach since parameterisation of ν_ eff=ν+ν_η appears only as an undetermined parameter. The parameterisation is significantly distinguished from the conventional tuning of model constants. The question remains as “how to determine the parameter ν_ eff (or ν_η) regarded as a spatial distributed variable?”. 
  
  With the paramaterised NSEs, assuming only limited data are available on a set 𝒳∈Ω∖Ω^' (cardinality 𝒩=|𝒳|) of finite scattered measurement locations, we are interested in the missing flow dynamics on Ω^' where the specified point set 𝒳^' with cardinality 𝒩^'=|𝒳^'| is chosen to represent the dynamics. On 𝒳, the set of measured averaged/filtered 𝐮 is denoted by 𝒮̅_𝒳 and on Ω^', the set of missing averaged/filtered 𝐮 is denoted by  𝒮̅_𝒳^'. Further, the undetermined ν_ eff appears as a spatial variable defined on Ω, and will be inferred later. Here, the target is to approximate the functional relation (t,x,y,z)↦ (u,v,w,p,ν_ eff) by physics-informed neural network (t,x,y,z)↦ (e_u,e_v,e_w,e_ div). The parameterised NSEs are encoded in e_u, e_v, e_w, and e_ div, which represent the residuals of the momentum equations and  the continuity equation, respectively. To overcome numerical stiffness that lead to unbalanced back-propagated gradients during model training, a recently developed learning rate annealing algorithm is adopted <cit.>. The physics-informed residuals e_u, e_v, e_w, and e_ div are minimized with the mean squared error loss function on the time set 𝒯
  
      Loss=𝔼_𝒯[ MSE_ Data+ MSE_ PDEs]
  
  where
  
      MSE_ Data=𝔼_𝒳[𝐮(t;𝒳)-𝐮_(t;𝒳)^2],
  
  and
  
      MSE_ PDEs=𝔼_𝒳̂[e_u^2+e_v^2+e_w^2+e_ div^2].
  
  Here 𝒳̂∈Ω. The operator 𝔼_#[·] means the average on the set `#' in a distribution sense. 𝔼_𝒯[·] is an identity operator for stationary modellings. Generally, the location cluster can be  generated in a probability sense. Once the above minimisation problem is solved, the extra variables p and ν_ eff can be discovered. 
  
  
  
  § NUMERICAL CONFIGURATION
  
  The diagram of the physics-informed neural network is illustrated in Fig. <ref>. We here treat the parameter ν_η as an output of the deep neutral network. During the training process, the required weights of neural network are determined by the loss function formed by the residual of the governing equations. Once the training convergence is achieved, the variables u, v, w, p and ν_η can be inferred. The training datasets were generated by computational fluid dynamics (CFD) and the open-source software OpenFOAM <cit.> was used to implement required computations. In two-dimension, only values of the velocity components u and v at mesh nodes are considered as training data. 
  
  To check the ability of the method, we considered three different cases, related to the parameterised governing equations of RANS, URANS and SAS for two different turbulent flows, respectively. SAS denotes “scale-adaptive simulation” <cit.>, and typically allows simulation of unsteady flows with both RANS and LES content. It should be noted that only regional velocity is used as training data, while velocity, pressure and artificial viscosity in the entire domain are inferred. 
  
  
  
  
  
  
  
  
  
  
  Two flows, including a steady turbulent flow over a back-ward facing step (RANS) and an unsteady turbulent flow past a cylinder (URANS and SAS), were involved. For the steady one, there are 1.2× 10^4 cell nodes in total, while the random sampling nodes are 3× 10^3 for training data (on Ω∖Ω^') and 4.3× 10^3 constraints of the parameterised governing system (on Ω), respectively. A neural network with 6 hidden layers and 50 neurons per layer is considered. 2×10^5 epochs are used. For the unsteady flow past a cylinder, there are 351 snapshots and 5.1× 10^5 cell nodes on every snapshot, thus 1.7901× 10^8 in total. The random sampling nodes are 1.2× 10^7 and 1.4× 10^7 respectively, on all time snapshots. Corresponding neural network has 10 hidden layers and 96 neurons per layer. The learning rate is gradually decreased from 1.0× 10^-3 to 1.0×10^-6 with 400 epochs at each stage. The Adam optimizer is adopted for all cases. 
  
  
  
  
  § RESULTS
  
  
  
  We first consider a steady turbulent flow over a backward facing step, experimentally studied by Pitz and Daily date back to 1981 <cit.>. The Reynolds number is 5× 10^5 based on the step height. The SST k -ω model was adopted to generate the data. Here SST denotes “shear-stress transport” <cit.>. The contours of velocity components along the streamwise and vertical directions are given in Figs. <ref>(a) and <ref>(a), respectively. It is assumed that in the dash-lined rectangular box, denoted by Ω^', the flow information  therein is missing and thus not used as training data. 
  
  After the neural network is trained with the data outside Ω^', together with constrains of the governing equations, the flow variables in the whole domain Ω are inferred. In Figs. <ref>(b) and <ref>(b), the inferred velocity components by the physics-informed deep learning are given.  It is clear that in the box region, we  get the similar  velocity fields as illustrated in Figs. <ref>(a) and <ref>(a), demonstrating that the averaged flow dynamics is mimicked correctly with the proposed parameterised NSEs. In Fig. <ref>, the inferred pressure field is in good agreement with the reference one from the SST k-ω model. It is interesting, on the other hand, to compare the inferred artificial viscosity ν_η with eddy viscosity ν_t empirically determined by the SST k-ω model. Although they have different meaning thus are different in both magnitude and spatial distribution, as shown in Fig. <ref>, their distributions are related to the flow structure and both are relatively large in the region with separation bubble. 
  
  
  
  
  
  
  
  
  
  For the same steady turbulent flow, we specially consider a case where the parameter ν_η is removed and thus the original NSEs are used as a physics constraint during the learning. The energy contours of the inferred velocity field 𝐮_ NSEs and the `perturbation' field (𝐮_ NSEs-𝐮̅) are given in Fig. <ref>. Special structures are observed around the tip of the separation bubble.  Compared with the solution based on the parameterised NSEs, this significant inconsistency provides some potential information about the region where the most unstable modes or the energetic turbulent structures emerge around the flow separation region.
  
  
  
  As a further demonstration,  we consider the time-dependent turbulent flow past a cylinder. The Reynolds number is, Re = u_in D/ν = 10^5, where u_in and D are the inlet velocity and diameter of the cylinder, respectively. The reference flows are generated respectively by the SST k -ω and SST-SAS turbulence models. It should be emphasized that, unlike other traditional RANS models (e.g. the SST k -ω model), the SST-SAS model has a pure RANS nature but achieves LES behavior. With this 
  time-dependent turbulent flow, we demonstrate that the proposed methodology has a great adaptability to turbulence models for unsteady flow dynamics. In practice, the reference flow data obtained by the SST-SAS and SST k -ω turbulence models can be regarded as the “filtered" experimental data. 
  
  It is assumed, again, that in the system there exists a local region where the flow dynamics is missing, as indicated by a dashed rectangular box in Fig. <ref>.  In order to employ the established methodology, we consider a series of flow field snapshots obtained by the SST-SAS model, and only a part of the velocity field beyond the dashed-rectangular box on every snapshot is employed for training. In Fig. <ref>(Mid), the reconstruction 
  of the missing flow dynamics at t=11.96 in the dashed-rectangular box is presented. Surprisingly, compared with Fig. <ref>(Top),  the reconstructed missing flow  coincides with the reference flow  very well, even with complex vortex structure in the box. Qualitatively, we show the differences between the reference fields and the inferred ones in Fig. <ref>(Bottom). Relative low deviation is observed. Meanwhile, the profiles of the lift and drag coefficients are given in Fig. <ref>. The inferred C_L profile is consistent with the reference one.  However, there exist some differences between the inferred C_D and the reference  one. As observed from the comparison, the unsteadiness (or “irregularity” ) of the C_D profile from the inferred data is reconstructed fairly well. Further, we show the distribution of the viscosity in Fig. <ref>. It is observed that the patterns of the viscosity from both the SST-SAS and neural network, are consistent with the vortex shedding pattern, but with different magnitudes. 
  
  
  
  
  
  
  
  Accordingly, for the flow past a cylinder, instantaneous results at t=7 with the SST k -ω model are given in Figs. <ref>-<ref>. We see that using the methodology with the physics-informed deep learning, the missing flow dynamics 
  whose location is assigned arbitrarily, is recovered properly with good accuracy. 
  
  
  
  
  
  
  
  With the cases investigated, we demonstrate that when the parameter ν_η is introduced as an output variable of the neural network, the parameterised governing system of flow dynamics has a strong adaptivity to approximate the data from the turbulence models. Significantly, the missing data in a randomly specified region can be inferred by the physics-informed deep learning. Given the reality that reasonable accurate velocity and pressure are predicted by the methodology, the inferred ν_η indicates that the eddy viscosity in the conventional turbulence models probably has an over-dissipation effect on the flow structures and a different spatial distribution from the inferred one. This discovery implies that the physics-preserved data driven techniques may have a transformative influence on the turbulence closure modelling.
  
  
  
  
  
  § CONCLUSION AND DISCUSSION
  
  The methodology proposed in this paper provides a way to explore the missing flow dynamics, which is well validated by the data from the turbulence models. We show that from the single parameterisation of the Navier-Stokes equations, the feasibility and potential of the method are significant for the “averaged” or “filtered” data, no matter obtained experimentally or numerically. Surprisingly, the current methodology allows to use the parameterised governing equations without extra modellings, which is shaping a perspective in exploring the dynamics beyond the description of the original governing system, although the deviation of exploring missing dynamics exists. From this point of view, the parameterisation, in fact, provides a solid foundation of accurately describing practical complex flow dynamics, which may not be accurately described by the traditional turbulence modellings where the artificial viscosity ν_ t needs to be further tuned.
  
  The proposed methodology provides a new way for further calibrations of the turbulence closures, which is probably difficult for traditional case-by-case machine learning with simply tuning the scalar modelling parameters. As indicated from the theory of turbulence modelling, due to the significance of the parameter ν_η, the proposed method is naturally termed as “data-driven scale-adaptive turbulent structure recovering”.
  
  Although only 2D incompressible flows were considered as examples in this work, the proposed methodology can be extended to 3D incompressible or compressible flows. While learning from the parameterised Navier-Stokes equations is successful in coping with the traditional fluid dynamics problems, this, in turn, implies generalisation of the employed methodology could enable exploration and exploitation in a broad range of complex systems, e.g. weather prediction, ocean recirculation and biofluid mechanics.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  § ACKNOWLEDGMENTS
  
  The authors would like to acknowledge support from National Numerical Wind Tunnel Project (Grant NO. NNW2019ZT4-B09) and the National Natural Science Foundation of China (Grant NOs. 91852106).
  
  
  
  
  § DATA AVAILABILITY
  
  
  The data that support the findings of this study are available from the corresponding author upon reasonable request. 
  
  
  
  
  
  § REFERENCES
  
  plainnat
  
  
  
  
  
  