Data-driven approximations of dynamical systems operators for control
Eurika Kaiser J. Nathan Kutz Steven L. Brunton


Data-driven approximations of dynamical systems operators for control





Eurika Kaiser Mechanical Engineering, University of Washington, Seattle, WA, 98195, eurika@uw.edu
J. Nathan Kutz Applied Mathematics, University of Washington, Seattle,
WA, 98195 kutz@uw.edu
Steven L. Brunton Mechanical Engineering, University of Washington, Seattle, WA, 98195, sbrunton@uw.edu





*
    Eurika Kaiser, J. Nathan Kutz, and Steven L. Brunton
    December 30, 2023
========================================================



*
The Koopman and Perron Frobenius transport operators are fundamentally changing how we approach dynamical systems, providing linear representations for even strongly nonlinear dynamics.  

Although there is tremendous potential benefit of such a linear representation for estimation and control, transport operators are infinite-dimensional, making them difficult to work with numerically.  

Obtaining low-dimensional matrix approximations of these operators is paramount for applications, and the dynamic mode decomposition has quickly become a standard numerical algorithm to approximate the Koopman operator.  

Related methods have seen rapid development, due to a combination of an increasing abundance of data and the extensibility of DMD because of its simple framing in terms of linear algebra.  

In this chapter, we review key innovations in the data-driven characterization of transport operators, providing a high-level and unified perspective.  

We emphasize important recent developments around sparsity and control, in particular, and discuss emerging methods in big data and machine learning.  	



The Koopman and Perron Frobenius transport operators are fundamentally changing how we approach dynamical systems, providing linear representations for even strongly nonlinear dynamics.  

Although there is tremendous potential benefit of such a linear representation for estimation and control, transport operators are infinite-dimensional, making them difficult to work with numerically.  

Obtaining low-dimensional matrix approximations of these operators is paramount for applications, and the dynamic mode decomposition has quickly become a standard numerical algorithm to approximate the Koopman operator.  

Related methods have seen rapid development, due to a combination of an increasing abundance of data and the extensibility of DMD based on its simple framing in terms of linear algebra.  

In this chapter, we review key innovations in the data-driven characterization of transport operators for control, providing a high-level and unified perspective.


We emphasize important recent developments around sparsity and control, and discuss emerging methods in big data and machine learning.  

































§ INTRODUCTION















Data-driven modeling using linear operators  has the potential to transform the estimation and control of strongly nonlinear systems.
Linear operators, such as Koopman and Perron-Frobenius operators, provide a principled linear embedding of nonlinear dynamics, extending the application of standard linear methods to nonlinear systems, and thus, significantly simplifying the control design and reducing the computational burden.


More broadly, data-driven discovery of dynamical systems is undergoing rapid development, driven by the lack of simple, or often entirely unknown, equations and the increasing abundance of high-fidelity data.
There have been recent successes in the discovery of functional representations of nonlinear dynamical systems, e.g. using evolutionary optimization techniques <cit.> and sparse optimization <cit.>.
However, control based on nonlinear equations is particularly challenging, becoming infeasible for higher-dimensional problems, lacking guarantees, and often requiring problem-tailored formulations.
In contrast, the emerging field of linear operators in dynamical systems seeks to embed nonlinear dynamics in a globally linear representation, providing a compelling mathematical framework for the linear estimation, prediction, and control of strongly nonlinear systems.
The rise of advanced data science and machine learning algorithms, vastly expanded computational resources, and advanced sensor technologies make this a fertile ground for the rapid development of data-driven approximations to these linear operators for control.
In this chapter, we review key innovations, discuss major challenges and promising future directions, and provide a high-level and unified perspective on data-driven approximations of transfer operators for control.



Data-driven system identification has reached a high degree of maturity. 
There exist a plethora of techniques that identify linear and nonlinear systems <cit.> based on data, including state-space modeling via the eigensystem realization algorithm (ERA) <cit.> and other subspace identification methods, Volterra
series <cit.>, linear and nonlinear autoregressive models <cit.> (e.g., ARX, ARMA, NARX, and NARMAX), and neural network models <cit.>, to name only a few. 
The reader is referred to <cit.> for an compressed overview of identification techniques for linear and nonlinear systems. 
In the machine learning community, manifold learning, e.g. locally linear embedding and self-organizing maps, and non-parametric modeling, e.g. Gaussian processes, have been proven to be useful for identifying nonlinear systems <cit.>. 
Most of these models are considered data-driven <cit.>, as they do not impose a specific model structure based on the governing equations. 
However, there is an increasing shift from black-box modeling to inferring unknown physics and constraining models with known prior information.
For instance, the recent sparse identification of nonlinear dynamics (SINDy) <cit.>, which has been extended to incorporate the effect of control <cit.>, is able to take into account known expert knowledge such as symmetries and conservation laws <cit.>. 
Learning accurate nonlinear models is particularly challenging as small deviations in the parameters may produce fundamentally different system behavior.
Importantly, it is possible to control many nonlinear systems using linear models. Examples include weakly nonlinear systems for which models are obtained based on a local linearization of the nonlinear dynamics at a reference point. 

However, the pronounced nonlinearities present in many applications generally require nonlinear control design.
The lack of robustness and stability guarantees except for special cases and the increased computational burden during the on-line phase, which becomes prohibitive for high-dimensional systems, restricts the application of nonlinear control to low-dimensional systems and requires specialized design techniques.
Encoding nonlinear dynamics in linear models through operator-theoretic approaches provides a new opportunity for the control of previously intractable systems. 


Linear embedding theory for nonlinear systems goes back to seminal works by B.O. Koopman in 1931 <cit.> and T. Carleman in 1932 <cit.>. 
Koopman showed that Hamiltonian dynamics can be described through an infinite-dimensional linear operator acting on the Hilbert space of all possible observables that can be measured from the underlying state. 
Closely related, Carleman demonstrated that systems of ordinary differential equations with polynomial nonlinearities can be represented as an infinite-dimensional system of linear differential equations.




Since the introduction of the so-called Carleman linearization, it has been applied to a wide range of problems <cit.>
including for the Lyapunov exponent calculation <cit.> and for finding first integrals <cit.>.

The emerging Koopman operator perspective provides an alternative direction and generalizes beyond polynomial systems.
The ultimate goal is to learn a globally linear embedding of the nonlinear dynamics such that powerful linear methods become immediately applicable and are useful in a larger domain. 

Of particular interest are the spectral properties of these operators that encode global information and can be related to geometrical properties of the underlying dynamical system. 
The potential to discover global properties of dynamical systems through operator-theoretic methods for diagnostic purposes, i.e. improving understanding of the underlying dynamics, has driven continued efforts to develop improved algorithms.
The confluence of big data, advances in machine learning, and new sensor technologies has further fueled the progress in data-driven methods, facilitating equation-free approximations of these operators. 
These efforts will be discussed below particularly in the context of system identification for control.
























































































We introduce the Koopman and Perron-Frobenius operators by considering the following autonomous nonlinear dynamical system


    d/dt(t) = ((t)),  (0) = _0

with ∈𝕏⊂ℝ^n, initial condition _0∈𝕏, and the flow is denoted by ^t so that (t)=^t(_0).



The nonlinear system (<ref>) can be equivalently described by infinite-dimensional, linear operators acting on observable or density functions.
The linearity of these operators is appealing; however, their infinite dimensionality poses issues for representation and computation and current research aims to approximate the evolution instead on a finite-dimensional subspace facilitating a finite-dimensional matrix representation <cit.>.




Let (𝕏, 𝔅,μ) be a measure space with state space 𝕏, σ-algebra 𝔅, and measure μ. 
The Koopman operator <cit.> is an infinite-dimensional linear operator that advances measurement functions f∈ L^∞(𝕏):


    f(t,_0) = U^t f(_0)= f(^t(_0)).



Any set of eigenfunctions of the Koopman operator span an invariant subspace, where the dynamics evolve linearly along these basis directions. 
Thus, the spectral analysis of the Koopman operator is of particular interest and spectral properties have been shown to be related to intrinsic time scales, geometrical properties, and long-term behavior of the underlying dynamical system <cit.>.
Koopman eigenfunctions ϕ() associated with a particular eigenvalue λ are special or intrinsic observables, which evolve linearly according to

    U^t ϕ(_0)= ϕ(^t(_0)) = e^λ tϕ(_0),

and which can generally be discontinuous <cit.>.
It can be shown <cit.> for continuously differentiable functions f with compact support, the function f(t,) := U^t[f]() satisfies the first-order partial differential equation (PDE):

    ∂/∂ tf(t,) =  F()·∇f(t,) = L_U f(),  f_0:=f(0,),

where L_U is the infinitesimal generator of the semigroup of Koopman operators {U^t}_t≥ 0, for which an exponential representation U^t = e^L_U t exists.
Smooth Koopman eigenfunctions can then be interpreted as the eigenfunctions of the generator and satisfy

    d/dtϕ() =  F()·∇ϕ()= λϕ().

As the Koopman operator evolves measurement functions, this perspective is particularly amenable to data-driven approaches.

The Koopman operator is dual to the Perron-Frobenius operator <cit.>, i.e. ⟨ P^tρ,f⟩ = ⟨ρ,U^t f⟩ for any ρ∈ L^1 and f∈ L^∞, and as a consequence these share certain properties. For instance, for measure-preserving systems these operators are unitary and exhibit the same point spectrum <cit.>.
The Perron-Frobenius operator propagates densities ρ∈ L^1(𝕏) and is defined as

    ∫_B  P^tρ() μ(d) =  ∫_^-t(B)ρ() μ(d)  ∀ B∈𝔅.

Further, 

    ρ(t,) = P^tρ()= ∫_𝔅δ(-^t(_0))ρ_0(_0) d_0,

where δ(-^t(_0)) represents the deterministic kernel. 
For an invertible system, this becomes P^tρ()= J^-t() ρ(^-t()),
where J^-t():=(d^-t()/d) is the determinant of the Jacobian of ^-t();
thus, the density varies inversely with the infinitesimal volume occupied by the trajectories.
For invertible and conservative systems we have P^tρ()= ρ(^-t()) with volume preservation div  = 0.
Eigenfunctions of the Perron-Frobenius operator satisfy 

    P^t ν() = J^-1() ν(^-1()) = e^λ tν().

Of particular interest is the
physical invariant measure μ^*(B) = μ^*(^-t(B)) for all sets B∈𝔅, which is stationary under the evolution of the flow.
The associated  invariant density 

Pρ^*() = ρ^*()

corresponds to an eigenfunction at eigenvalue 1, which describes the asymptotic behavior of the underlying dynamics. Control is often designed to alter the observed invariant measure or density.

Spectral properties of the Perron-Frobenius operator are related, e.g., to almost-invariant sets, meta-stable states, mixing properties, decay of correlations <cit.>. 



The infinitesimal generator of the semigroup of Perron-Frobenius operators { P^t }_t≥ 0 is given by the Liouville operator L_P <cit.>:

    ∂/∂ tρ(t,) = - ∇·( F() ρ(t,)) = L_P[ρ](),  ρ_0:=ρ(0,),

for continuously differentiable ρ with compact support and appropriate boundary conditions.
The Liouville equation (<ref>) describes how the flow transports densities in phase space and has a very intuitive interpretation as the conservation of probability or mass in terms of trajectories in the phase space. Alternatively, the evolution of the density may interpreted as propagated uncertainty of an initial state.
This is a first-order PDE that can be solved with the method of characteristics <cit.>. 
The invariant density satisfies  L_P[ρ^*]()=0 and is an eigenfunction at eigenvalue 0.












































































 
 























The aim of this chapter is to provide an overview of major developments and advances in data-driven control using operator-theoretic methods. 


The chapter is organized as follows:
In Sec. <ref>, the general control problem for a nonlinear system is formulated from an optimal control perspective. 
Control-dependent Koopman and Perron-Frobenius operators are introduced in Sec. <ref>. 
The main objective of the operator-theoretic approach is to find a linear representation of the underlying dynamical system. Data-driven system identification methods for the controlled nonlinear system are summarized in Sec. <ref>. 
Important aspects in control theory such as observability, controllability, state estimation, and control design in the operator-theoretic framework are discussed in Sec. <ref>.

The chapter is concluded in Sec. <ref> with a discussion on major challenges, open problems, and possible future directions of transfer operators approximations and their application for control.






§ CONTROL PROBLEM FORMULATION


Reformulating strongly nonlinear dynamics in a linear framework via transfer operators is appealing as it enables the application of powerful optimal and robust estimation and control techniques available for linear systems <cit.>.  
The formulation of an optimal control problem <cit.> appears in many applications, such as trajectory control in robotics or boundary layer stabilization in fluids, and has been considered widely in the context of Koopman operators for control <cit.>.

Optimization-based approaches for control provide a highly suitable framework for the control of nonlinear system, e.g. including constraints and flexible cost formulations. The optimal control problem can be stated as follows:


    min_∈𝕌   ∫_t_0^t_f  l[(t),(t),t]    d t +l_f[(t_f),t_f]


subject to

	
    d/dt(t)    = 𝐅(,),  (0) = _0
    (t)    =  (,)


and possibly additional constraints on states, measurements, control inputs or time. 
We consider the nonlinear system (<ref>) extended to include an external input ∈𝕌⊂ℝ^q, where 𝕌 is the space of admissible control functions (or sequences in discrete time), and assume continuously differentiable state dynamics 
F:𝕏×𝕌→𝕏.

The state  may not be fully accessible and instead a limited set of output measurements ∈𝕐⊂ℝ^p may be collected, which are prescribed by the measurement function :𝕏×𝕌→𝕐.
The nature of the solution of the optimization problem is determined by the choice of the terminal state cost l_f[·] and the running cost l[·].
The objective is to determine a control law or policy that minimizes the cost functional (<ref>). 

A nonlinear optimal control formulation for the system in (<ref>) can be established using dynamic programming <cit.>. 
This relies on Bellman's principle of optimality <cit.>
and leads to the Hamilton-Jacobi-Bellman (HJB) equation <cit.>, a nonlinear PDE for the globally optimal solution.
Solving this nonlinear PDE is computationally challenging and as a result, a variational argument and Pontryagin's maximum principle <cit.> is often used instead, leading to a set of coupled ordinary differential equations (ODEs), the Euler-Lagrange equations.
While this two-point boundary value problem is solvable for higher-dimensional problems in contrast to the HJB equation, it is still too computationally demanding in real-time applications for many nonlinear systems, e.g. in autonomous flight <cit.>.
Moreover, for high-dimensional systems, such as fluid flows, expensive direct and adjoint simulations render this approach infeasible, instead motivating the use of reduced-order models. 

Controlling nonlinear, and possibly high-dimensional, systems is generally computationally demanding and performance and robustness guarantees exist only for certain classes of dynamical systems. 
However, the control problem above simplifies considerably for quadratic cost functions and linear dynamics of the form

    d/dt =  + ,  (0) = _0,
 
where the matrix  may be obtained by a suitable linearization of the nonlinear dynamics (<ref>) around an equilibrium point or operating condition.

Considering linear dynamics, the optimization problem can be simplified tremendously and becomes solvable even for high-dimensional systems. 
In the simplest case, without any constraints, this reduces to solving an algebraic Riccati equation (ARE) and yields the linear quadratic regulator (LQR) <cit.>. 



Many problems may not permit a fully linear representation or the approximation may only be valid close to the linearization point. 
Moreover, models estimated from data may quickly become invalid, either due to the application of control or changing system parameters and conditions. 
For these cases, there exist adaptive and nonlinear variants of the formulation above. 
For instance, control-affine, nonlinear systems, whose governing equations can be factored into a linear-like structure, permit a state-dependent transition matrix () and actuation matrix () so that the ARE may be solved point-wise as a state-dependent Riccati equation (SDRE) <cit.>. The SDRE generalizes LQR for nonlinear systems, retaining a simple implementation and often yielding near optimal controllers <cit.>. 
Alternatively, multiple model systems can be considered, which usually consist of a large set of locally valid models, for which control is then determined either using a specific model selected based on some metric or by averaging/interpolating the control action from the model set <cit.>.

A widely adopted adaptive variant is model predictive control (MPC) <cit.>, which solves the optimal control problem over a receding horizon subject to the modeled dynamics and system constraints.
The receding horizon problem is formulated as an open-loop optimization over a finite time horizon. At each time step and given the current measurement, a sequence of future control inputs minimizing the cost J over the time horizon is determined.
The first control value of this sequence is then applied, and the optimization is re-initialized and repeated at each subsequent time step. This
results in an implicit feedback control law (_j) := _j+1(_j), where _j+1  is the first input in the optimized actuation sequence starting at the initial condition _j:=(t_j).
MPC is particularly ubiquitous in industrial applications including process industries <cit.> and aerospace <cit.>, as it enables more general formulations of control objectives and the control of strongly nonlinear systems with constraints, which are difficult to handle using traditional linear control approaches.

Linear models arising as approximations of the Koopman or Perron-Frobenius operators may be combined with any of these approaches. The potential benefit is the increased accuracy and validity as compared with linear models based on a local linearization and a reduced computational burden compared with schemes based on nonlinear models. Moreover, many data-driven approximation algorithms for these operators are easy to implement and efficient to compute as these are founded on standard linear algebra techniques. 





































§ CONTROL-ORIENTED TRANSFER OPERATORS




Controlled dynamical systems have been increasingly considered in the operator-theoretic framework, particularly for state estimation, to disambiguate dynamics from the effect of control, and for related problems such as optimized sensor and actuator placement.

While the Liouville equation has been examined for control purposes for a longer time <cit.>, and interestingly also in the context of artificial intelligence <cit.>, the control-oriented formulations of the Koopman and Perron-Frobenius operators have gained traction only recently <cit.>.
















 §.§ Non-affine control systems

We consider a non-affine control  dynamical system and  assume access to the full state =:

    d/dt(t) = 𝐅(,),  (0) = _0.

The flow associated with (<ref>) is referred to as the control flow and is denoted by ^t(,).  Skew-product flows, such as ^t(,), arise in topological dynamics to study non-autonomous systems, e.g. with explicit time dependency.
Thus, we consider the action of an operator in the extended state space 𝕏×𝕌.

The Koopman operator is defined as acting on the extended state:

    U^t f(,) = f(S^t(,),).

Here the inputs  evolve dynamically, e.g. by a prescribed exogenous behavior or a given state-dependent feedback control law = ().
If the inputs themselves are not evolving dynamically, this would reduce to U^t f(,) = f(S^t(,),0), 
in which case the inputs parameterize the dynamics <cit.>.
The spectral properties of U should then contain information about the unforced system with = 0.
Koopman eigenfunctions associated with (<ref>) satisfy

    U^t ϕ(,) = e^λ tϕ(,).

Assuming smooth dynamics and observables, the generator equation for the Koopman family is:

    ∂/∂ t f(t,,) = (,) ·∇_ f(t,,) + ·∇_ f(t,,).

As above, smooth Koopman eigenfunctions can be considered eigenfunctions of the infinitesimal generator satisfying

    d/dtϕ(,) = (,) ·∇_ϕ(,) + ·∇_ϕ(,)= λϕ(,).


Representing the system in terms of a finite set of observables or eigenfunctions generally requires a reformulation of the cost functional (<ref>) so that

    J =  ∫_t_0^t_f  l^K[(t),(t),t]    d t +l_f^K[(t_f),t_f],

where (t)=  f((t),(t)) describes the nonlinear transformation of the state  through a vector-valued observable f(,)=[f_1(,),…,f_d(,)]^T.

It may be of interest to directly control specific observables, e.g. Koopman eigenfunctions, that are associated with a particular physical behavior. 
If the state  is included as an observable, it is possible to transform the cost functional (<ref>) into (<ref>) by modifying l^K and l_f^K accordingly. 
However, there does not exist a finite-dimensional Koopman-invariant subspace explicitly including the state, that is topologically conjugate to a system representing multiple fixed points, limit cycles, or more complicated structures <cit.>.
Nevertheless, it may be possible to obtain a linearization in the entire basin of attraction for a single fixed point or periodic orbit <cit.>.
Further, Koopman eigenfunctions may be considered as observables and the state may be recovered via inversion, e.g. approximated from data using multidimensional scaling <cit.>.



Analogously, we can consider the Perron-Frobenius operator acting on densities

    P^t ρ(,) = J^-1()ρ(S^-t(,),),

for which eigenfunctions satisfy

    P^t ν(,) = e^λ tν(,).

The  control-oriented Liouville equation describing the transport of densities under the influence of an exogenous input is given by: 

    ∂/∂ tρ(t,,) = - ∇_·( (,) ρ(t,,)) - ∇_·( ρ(t,,)),  ρ_0(,) :=ρ(0,,),

where the  initial condition is, e.g., ρ_0(,) = δ('-)δ('-), a point-mass at ' and ' for deterministic dynamics.
This assumes the conservation of probability in the state-action space. The first term on the right-hand side describes the local change in density due to the flow, while the second term describes the local change due to exogenous inputs.

We are often interested in changing the long-term density ρ(t,).
The objective is then to determine inputs  so that ρ(t,) becomes close  to a desired target density ρ^T() over some time horizon [0,t_f] or in the limit t_f→∞, which can be interpreted as steering a density of initial conditions or particles governed by the underlying dynamical system (<ref>).
Assuming further that the inputs do not evolve dynamically, i.e. u̇ = 0, we have:

    ∂/∂ tρ(t,) = - ∇_·( (,) ρ(t,)),

which is considered in the seminal works of R. Brockett <cit.>, particularly in the context of ensemble control. 
The finite-time control objective may be formulated as

    J = ∫_t_0^t_f ∫_𝕏[ρ(t,) - ρ^T()]l^P_x() d + l^P_u[(t),t]dt

measuring the weighted deviation from the desired density and penalizing control input and time.
Ideally, the controlled system has a unique equilibrium density ρ^T(), corresponding to the sole eigenfunction of the Perron-Frobenius operator with eigenvalue 1, with all others decaying exponentially.
In <cit.>, a general cost functional of the following form is proposed:

    J = ∫_t_0^t_f∫_𝕏ρ(t,)l(,)d dt+ ∫_𝕏( ∂/∂)^2d + ∫_t_0^t_f(∂/∂ t)^2 dt,

where the first term evaluates the average performance of the system at the current time, the second term penalizes high gain feedback, and the third term may promote linearity of the control law.  

In general, different control perspectives can be assumed: (1) actively modifying the observable or density function, or (2) directly controlling the trajectory informed by the evolution dynamics of the observable or density function, e.g. ensemble control in a drift field.
An illustrative schematic is provided in Fig. <ref> using a particle in a field analogy.









 §.§ Affine control systems

A control-affine dynamical system is given by

    d/dt(t) = 𝐅()+𝐆(),  (0) = _0,

where 𝐆():=∑_i g_i( x)u_i represents mixed terms in x and u.  

Above we sought  a representation on the extended state (,), in which case the spectral properties of the associated operators will contain knowledge on the unforced system with = 0. Alternatively, we can consider operators associated with the unforced dynamics  and how these are affected by the external control input, rendering the representation non-autonomous.
Then, additional terms will arise and inform how control affects observables, densities, and eigenfunctions:

    ∂/∂ t f(t,) = (()+())·∇ f(t,)   =()·∇ f(t,) + ()·∇ f(t,),

where ∇:=∇_.
We assume Koopman eigenfunctions are associated with the unforced dynamics ().
Then, the smooth Koopman eigenfunctions satisfy

    d/dtϕ() = λϕ() + ∇ϕ()·(),

which renders the dynamics governing the eigenfunctions generally nonlinear.
Only in the special case, where () is a linear function of the state , this remains linear. The equation becomes bilinear for quadratic (), for which a rich control literature exists <cit.>.

The Liouville equation for the density becomes:

    ∂/∂ tρ(t,) = - ∇·( (()+()) ρ(t,)) 
    = - ∇·( (() ρ(t,)) - ∇·(()ρ(t,))

and correspondingly eigenfunctions satisfy:

    d/dtν() = λν() - ∇·(()ν()).

The controlled Liouville equation has also been examined in other contexts, where a scalar observable is advected with the flow, e.g. concentration of chemical components or temperature distribution, in cases where diffusion is assumed negligible.

It may be possible that the controller directly injects or removes content to modify the density, e.g. 
for carbon removal strategies in the atmosphere or oil spill mitigation in the ocean.
Then, (<ref>) is modified to 
 
    ∂/∂ tρ(t,) = - ∇·( (() ρ(t,)) + ∑_i=1^p  g_i() u_i,

 where g_i() is a set of functions prescribing the spatial impact of the actuators and the vector field  remains constant in time.

































§ SYSTEM IDENTIFICATION


In this section, we outline data-driven strategies to learn control-oriented representations of dynamical systems in the operator-theoretic framework.
Most approaches rely on dynamic mode decomposition for the Koopman operator and the classical Ulam's method for the Perron-Frobenius operator.
Emerging techniques such as machine learning and feature engineering aim to provide generalizations for systems of higher complexity, while exploiting and promoting sparsity often targets interpretability and efficient computation.
A general overview of algorithms for data-driven approximation of the Koopman and the Perron-Frobenius operators have been provided in <cit.>, which will not be repeated here. 
Here, we discuss approaches that specifically include the effect of control or external parameters. 















 §.§ Koopman operator




























System identification for  the Koopman operator can be broadly divided into three main directions: (1) linear models as approximations to the Koopman operator describing the evolution of a, usually large, set of observable functions, (2) (bi)linear models for Koopman eigenfunctions, which constitute a Koopman-invariant subspace, and (3) linear models parametrized by the control input. 
We assume a collection of data tuples (_k,_k',_k), where the state _k:=(t_k)∈ℝ^n, the time-shifted state _k':=(t_k+Δ t)∈ℝ^n, and inputs _k:=(t_k)∈ℝ^q are sampled at discrete times t_k, k=1,…,m. 
We define a vector-valued observable function f() = [f_1(),f_2(),…,f_d()]^T with f:ℝ^n→ℝ^d evaluated on data, which is generally a nonlinear transformation of the state so that =  f() embeds the state in a feature space of possibly higher dimension, i.e. d≫ n.


Dynamic mode decomposition with control:  
Building on DMD, the control-oriented extension DMD with control (DMDc) was proposed to disambiguate the effect of control from the natural dynamics in high-dimensional systems <cit.>. 

Assuming the availability of full-state measurements, observables are taken as linear measurements of the state, i.e. f =.

Then, the dynamical behavior of coordinates  are modeled as (t) = (t) + (t) or 

    _k+1 = _k + _k
 
in discrete time, which makes standard linear control techniques readily applicable.

The dimensionality of the problem is determined by the dimension of the state. However, many systems evolve on a low-dimensional attractor so that the singular value decomposition (SVD) can be used to solve the problem efficiently. Further, DMDc has been combined with compressed DMD <cit.> for compressive system identification <cit.>.
While DMDc is feasible for high-dimensional systems with n≪ m,  such as fluids that may have millions or billions of degrees of freedom, linear measurements are known to not be sufficiently rich to capture strongly nonlinear behavior, motivating the use of nonlinear observables.

Nevertheless, recent work <cit.> indicates that even models with relatively little predictive power may still be sufficient for model predictive control schemes (see also Sec. <ref>). 


Extended DMD with control: Extended DMD (eDMD) has been proposed as a generalization of DMD employing a dictionary of nonlinear functions as observables <cit.>. DMD is then equivalent to eDMD with the special set of observables corresponding to linear monomials, f() = [x_1,x_2,…,x_n]^T. 

In domains where such a local linearization would be insufficient, more complex basis functions, such as high-order monomials, Hermite polynomials, or radial basis functions, can be employed. 
The number of observables typically exceeds the state dimension, i.e. d≫ n; however, the algorithm can be reformulated to scale with the dimension of samples instead of the dimension of features/states.
eDMD has also been extended for nonlinear system identification, which we refer to as eDMD with control (eDMDc) <cit.>, and been combined with MPC <cit.>. 
By transforming the system so that the dynamics evolve linearly, linear MPC methods become readily applicable, which require solving a convex quadratic program instead of a non-convex program as in nonlinear MPC.  In <cit.> the MPC optimization is further reformulated to scale with the dimension of the state n and not with the number of observables d, which is advantageous if d ≫ n. 
However, the quality of the resulting model and its prediction is highly sensitive to the choice of function basis. 






Time-delay coordinates and DMDc: Often we do not have access to the full state, but instead can only measure a single or few variables. Then time-delay coordinates, i.e. f((t)) = [x(t),x(t-1τ),x(t-2τ),…,x(t-(d-1)τ)] with time delay τ, have been proven to be  useful for analyzing and modeling dynamics.
The Taken's embedding theorem <cit.>

provides conditions under which the original system can be faithfully reconstructed in time-delay coordinates, providing a diffeomorphism between the two systems. 
Recently, time-delay coordinates have been shown to provide a Koopman-invariant subspace and used to model unforced systems with bimodal or bursting behavior as linear systems closed by an intrinsic forcing term <cit.>. 
This formulation is achieved by performing DMD on the delay coordinates arranged in a Hankel matrix <cit.>.  
Hankel matrix based system identification has been used for decades in the eigensystem realization algorithm (ERA) <cit.>. 




Assuming sample pairs (x_k,u_k) of a single long trajectory x(t) generated by a continuous system with single control input u(t), 
the vector-valued observable is given by _k =  f(_k) = [x_k, x_k-1, x_k-2,…, x_k-d_1+1] where f_i(x_k)=x_k-(i-1) and _k =  [u_k, u_k-1, u_k-2,…, u_k-d_2+1].
The embedding dimensions d_1=d_2=d are assumed equal for simplicity. 
The system is then given by

	
    _k+1 = _k + _k
    _k = [ 1 0 … 0 ]_k


where the state is recovered from the first component of .
The control matrix  must satisfy a lower triangular structure to not violate causality.
If the number of time-delay coordinates is large, an SVD can be applied and the model is built on the first r eigen-time delay coordinates. 
It can be advantageous to embed the full state  if available instead of just a single component, as more information is used to improve prediction accuracy.
In Eq. (<ref>) the control history appears as a separate vector of control inputs; it can also be useful to augment the state with the past control values so that the current actuation value appears as single control input:

	
    _k+1 = 
    	[      _k;   u_k-1;       ⋮; u_k-d+1 ]_k+1   = 
    	[[                    _[2,d];            0 [ 0 0; 0 I ] ]]
    	[      _k;   u_k-1;       ⋮; u_k-d+1 ]_k + 
    	[ _[1];    1;    0;    ⋮;    0 ]
    	u_k
    _k   = [ 1 0 … 0 ]_k,


where I denotes the d-2× d-2 identity matrix and _[a,b] contain columns a through b of . 



Control in eigenfunction coordinates: Eigenfunctions of the Koopman operator provide a natural set of intrinsic observables as these themselves behave linearly in time and correspond to global properties of the system. 
Particularly, control affine systems Eq. (<ref>) have been studied in this context. 
The general approach relies on estimating eigenfunctions associated with the autonomous system and then identifying how these are affected by the control input.  
The observables are then given by a set of eigenfunctions =  f() = [ϕ_1(), …, ϕ_r()].  Their continuous-time dynamics are given by

    = (()+())·∇_ f() =  + () ,

where := ()·∇_ f() =  := diag(λ_1,…,λ_r)
and ()  := ()·∇_ f().
For =  0, this constitutes a Koopman-invariant subspace for the unforced system; otherwise, the control term () prescribes how these eigenfunctions are modified by .
The advantage of this formulation is that the dimension of the system scales with the number of eigenfunctions, and we are often interested in a few dominant eigenfunctions associated with persistent dynamics. 
This formulation has been examined for state estimation and observer design <cit.> and data-driven control <cit.>.
In general, the eigenfunctions can be identified using eDMD, kernel-DMD or other variants, and the model (<ref>) may be well represented as long as their span contains (), (), and  <cit.>.  
However, the eigenfunctions obtained from DMD/eDMD may be spurious, i.e. they do not behave linearly. 
For instance, noise can produce corrupt, physically irrelevant but energetically important, unstable eigenfunctions. 
An alternative <cit.> seeks a functional representation of the eigenfunctions in a dictionary of basis functions Θ() = [θ_1(),θ_2(),…,θ_p()] so that ϕ()≈∑_i=1^p θ_i()ξ_i = Θ()ξ.
A sparsity constraint on the vector of coefficients ξ∈ℝ^d  can then be used to identify an analytical expression of the eigenfunction. This approach is restricted to smooth eigenfunctions in the point spectrum of the Koopman operator and results in an optimization problem to find a sparse vector in the nullspace of the matrix (·∇_Θ() - λΘ())ξ = 0 for a given eigenvalue λ. 
This requires an accurate estimation of the time derivative of  and is sensitive to noise. The control term () can either be determined from the gradient of the identified eigenfunctions for given () or identified directly <cit.> for unknown () by similarly expanding it in terms of a dictionary, () ≈∑_i=1^q ψ_i() η_i.
The approach can be further extended to nonaffine control systems, for which control-dependent eigenfunctions may be considered, using a library on the extended state, i.e. Θ(,).
This data-driven characterization has been combined with SDRE <cit.> and MPC <cit.> for controlling the underlying dynamical system. 
The cost functional generally needs to be formulated in eigenfunction coordinates as discussed in the previous section.



















Parameterized linear models:  Another class of linear parameter-varying models have been considered to disambiguate the approximation of the Koopman operator of the unforced system from the effect of exogenous forcing <cit.> and for model predictive control of switched systems <cit.>. 
In particular, a set of linear models parametrized by the control input is considered as an approximation to the non-autonomous Koopman operator:

    _k+1 =  () _k.
 
For discrete-valued inputs _i∈{_1,…,_q }, a finite set of matrices defined as {_i:=(_i)}_i=1^q may be estimated, in the simplest case with DMD or a variant by determining these for each discrete control input separately.
This replaces the non-autonomous Koopman operator with a set of autonomous Koopman operators,  each element associated with a constant control input, as considered in <cit.>.
A more robust way that generalizes to continuous input data has been proposed in <cit.>, where () is expanded in terms of basis functions ψ_i:𝕌→ℝ as in eDMD for the state variable, so that () = ∑_k=1^dψ_k() _k. Then, for a given value of , matrix  is constructed using coefficients _k and its spectral properties can be examined. Since eDMD may be prone to overfitting, the estimation problem can be regularized using the group sparsity penalty <cit.>.   

Determining an optimal control sequence for Eq. (<ref>) is a combinatorial problem, which can be efficiently solved for low-dimensional problems using iterative algorithms from dynamic programming <cit.>.
In <cit.>, the MPC problem for the sequence of control inputs is transformed into a time switching optimization problem assuming a fixed sequence of consecutive discrete control inputs. The model can further be modified as a bilinear continuous, piecewise-smooth variant with constant system matrices, which does not suffer from the curse of dimensionality and allows for continuous control inputs via linear interpolation <cit.>. 













 §.§ Perron-Frobenius operator


A classical method to approximate the Perron-Frobenius operator is the Ulam-Galerkin method, a particular Galerkin method where the test functions are characteristic functions. Given an equipartition of the phase space into disjoint sets {B_1,…,B_d} and data, the action of the operator can be approximated as 

    _ij^τ = card( _k |_k ∈ B_j ∧^τ(_k) ∈ B_i)/card(_k ∈ B_j),
 




where ^τ is the flow map and τ is the time duration.
The τ-step stochastic transition matrix acts on a discrete probability vector =[p_1,…,p_d]^T and satisfies the properties ∑_j=1^dP_ij=1 and  0≤ P_ij≤ 1. A widely used software package based on set-oriented methods is GAIO <cit.>. 
Each element B_i can be associated with a distinct symbol. The dynamics are then propagated on the partition and can be analyzed in terms of the sequence of symbols.
The approach has been generalized to PDEs <cit.>, i.e. where the evolution of a pdf p(,t) of a spatiotemporal vector field (,t) is of interest. After expanding the vector field (,t) = ∑_i a_i(t)_i(), e.g. using POD, Ulam's method can be applied to the a_i's.
Other data-driven methods to approximate the Perron-Frobenius operator include blind source separation, the variational approach for conformation dynamics (VAC) <cit.> 

and DMD-based variants exploiting the duality between Koopman and Perron-Frobenius operators, e.g. naturally structured DMD (NSDMD) <cit.> and constrained Ulam dynamic mode decomposition <cit.>. 

Data-driven control based on the Perron-Frobenius operator appears more challenging as it relies on good statistical estimates requiring large amounts of data.
In practice, methods often use a Monte-Carlo sampling to estimate the transition probabilities, which suffers from the curse of dimensionality. 
However, it is particularly appealing as it provides a natural framework to incorporate uncertainties and transients, allows the control of ensembles and mixing properties of the underlying dynamical system, and yields a more global controller.
















Parametrized models: 
Identifying a probabilistic model disambiguating the unforced dynamics and the effect of actuation is commonly realized by a parametrized representation.
Incorporating the effect of control can be achieved via a simple extension of Ulam's method by considering a discretized control input that parametrizes the transition matrix in (<ref>): 

    _ij^u_l = card( _k |_k ∈ B_j ∧^τ(_k,_l) ∈ B_i)/card(_k ∈ B_j),
 
where each (_l):=_ij^u_l satisfies the normalization and positivity properties. 
The dynamics of the probability vector can then be expressed as

    _k+1 = (_l) _k.
 
Approximations for (<ref>) have been obtained employing set-oriented methods <cit.> and from NSDMD and using Gaussian radial basis functions <cit.>, which are then used to derive optimal feedback control strategies.
The model (<ref>) can be derived from a discretization of the Liouville equation in space, time, and control <cit.> and represents a control-dependent Markov chain.
The corresponding control problem can be associated with a Markov decision process (MDP) and appears in discrete optimal control <cit.>. 
Therein, it is used as a model to predict the expectation of the value function in dynamic programming, without reference to the Liouville equation. 
Building on these ideas, cluster-dependent feedback control has been developed by examining the spectral properties of the control-dependent transition matrix family <cit.>, where state discretization of a fluid from high-dimensional snapshot data is achieved using POD and k-means clustering.


Multiplicative models: 
Alternatively, the effect of control may be encoded in a separate stochastic matrix ^u approximating the action of a stochastic kernel: 

    _k+1 = ^u_k,
 
which recovers the unforced dynamics when ^u =  I is the identity matrix.
This model appears, e.g., in the context of optimizing mixing properties <cit.>, where  represents advection via the Perron-Frobenius operator and  ^u represents the discretized diffusion kernel to be optimized.
In <cit.>, limitations of nonlinear stabilization are examined when ^u = ^1+K encodes a state feedback law. 











Additive models: It may also be possible to formulate additive models, which exhibit some similarity to traditional state-space models, of the form

    _k+1 = (  + ^u)_k,
  
where additional constraints are necessary, so that + ^u satisfies the positivity and conservation constraints from probability. The matrix ^u may be interpreted as a disturbance to the unforced transition probabilities . 
It may also be possible to represent ^u as ^c_k, where ^c is a constant matrix and _k is the control input.

A similar formulation to (<ref>) incorporating control appears from a state-discretization of the controlled Liouville equation as outlined in <cit.>, resulting in a discrete-state, continuous-time Markov model.
Variations of this type of model have been used to study optimal response of Markov chains <cit.> and to control chaos <cit.>.


Remark: 
Until recently, the focus on control in the  Perron-Frobenius and Liouville operator framework has been mostly analytic and theoretical. Further, most problems studied have been low-dimensional. 
On the other hand,  there exists an extensive literature on probabilistic models for control, 
which can benefit the operator-theoretic framework but which have not been fully explored yet.
Examples include Hidden Markov models, see <cit.> in the context of the Perron-Frobenius and Fokker-Planck operators, and partial-observation Markov decision processes (POMDP) <cit.>.





































 §.§ Numerical example


We compare the effectiveness of model predictive control for different DMDc-based models, which differ in their choice of dictionary functions. 
The van der Pol oscillator is considered as an illustrative example:

    d^2 x/d t^2 - μ(1-x^2)d x/d t  + x = u

with μ = 0.2 and where u is the external control input.
In the following, the state vector is defined as :=[x_1,x_2]^T=[x,dx/dt]^T.
The training data consists of 200 trajectories in the box [-6,  6]× [-6,  6] integrated until T=1 with timestep Δ t=0.05, i.e. 4000 sample points in total. The forcing u(t) = 5 sin(|ω_1| t)sin(|ω_2| t), where ω_i∼𝒩(0,10), is applied for the identification task.
The models under consideration include DMDc on the state , eDMDc using monomials up to order 5, and delayDMDc with embedding dimension d=5, as compared in Fig. <ref>. 
For the control optimization, weight matrices Q = ([ 1 0; 0 1 ]), R_u = 0.1, and R_Δ u=0.1, and input constraints -5<u<5, -50<Δ u<50 are employed. The prediction/control horizon is T = 0.75= 15 Δ t. 

The models are evaluated on a test dataset that is different from the training set, visualized for a subset of trajectories in Fig. <ref>(a). 
The smallest error is achieved by eDMDc, followed by delayDMDc and DMDc, with the latter exhibiting a large deviation from the actual evolution.
MPC control results are displayed for μ = 0.2 in Fig. <ref>(b)-(c).
Three main observations can be made: 
(1) Despite the prediction inaccuracies, all three models are able to successfully control the system;
(2) best performance is achieved with eDMDc and delayDMDc, while delayDMDc generally performs slightly better; 
(3) the farther away the initial condition is from the fixed point to be stabilized, the worse the performance for DMDc (compare results in Fig. <ref>(d)).
It is important to point out that the specific control performance is highly sensitive to the chosen prediction horizon, weights, and the forcing for the training data; however, delayDMDc appears to be most robust. 
Observed here and in previous work <cit.>, superior prediction performance may be overrated in combination with predictive control schemes. The overall performance lies more in the robustness of MPC than in the specific model choice due to the repeated initialization with the updated measurement vector and the sufficiently short prediction horizon.










§ CONTROL THEORY IN THE OPERATOR-THEORETIC FRAMEWORK









In this section, we review control-theoretic advances using transfer operators and their connections to classical concepts in control such as stability analysis and observability.




 §.§ Stability Analysis




Analyzing the stability of dynamical systems is an important aspect of control theory, e.g. in order to develop stabilizing controllers. 
In particular, the Lyapunov function plays an essential role in the global stability analysis and control design of nonlinear systems. 


For stable, linear systems, the Lyapunov function is the positive solution of a corresponding Lyapunov equation. 
In contrast, finding and constructing a Lyapunov function poses a severe challenge for general nonlinear systems.
There has been important progress within the operator-theoretic context for nonlinear stability analysis as spectral properties of the operators can be associated with geometric properties such as Lyapunov functions and measures, contracting metrics, and isostables.

Some of this work relates back to a weaker notion of stability in terms of a density function as introduced by A. Rantzer <cit.>.
In particular, he introduces the concept of asymptotic stability in an almost everywhere (a.e.) sense, which is guaranteed through the existence of a density function.
Around the same time, it was shown that Lyapunov functions can be constructed using a suitable choice of polynomials, resulting in a linear problem formulation <cit.>.
These ideas were later combined in a convex, linear formulation based on the discretization of the density function via polynomials, to jointly synthesize the density function and state feedback.
The crucial connection of this line of work to the Perron-Frobenius operator was made by U. Vaidya and P. Mehta <cit.> demonstrating that these ideas of duality and linearity can be expressed using methods from ergodic theory, and the Lyapunov measure represents the dual of the Lyapunov function, including its relationship to the Koopman operator. 
Spectral properties, specifically invariant sets, are then used to determine stability properties. The Lyapunov measure is closely related to the density function introduced in <cit.> and captures the weaker a.e. notion of stability. 
Analogously to the Lyapunov equation, the solution to the Lyapunov measure equation provides necessary and sufficient conditions for a.e. stability of nonlinear systems. 
The Lyapunov equation and Lyapunov measure have been connected to the Koopman and Perron-Frobenius operator, respectively, e.g. providing an explicit formula for the Lyapunov measure in terms of the Perron-Frobenius operator, which generalizes stability analysis to invariant sets of nonlinear systems <cit.>.


Set-oriented methods have been used for solving the Lyapunov measure equation leading to a linear program <cit.>. 
A weaker notion of coarse stability has been further introduced to characterize the stability properties obtained from the finite-dimensional approximation.
A formulation for the infinitesimal generator of the Perron-Frobenius operator, the Liouville operator, has also been proposed and can be solved numerically using the finite-element method <cit.>.
In particular, for a stable fixed point ^*∈𝕏, a Lyapunov measure (or density) μ satisfies a.e.

    L_P [μ] ()<0  ∀≠^*,

which corresponds to ∇ (μ())>0 as originally introduced in <cit.>.
Thus, the Lyapunov measure (density) decays a.e. under the action of the Liouville operator and is related to its spectral properties.
These ideas have further been extended to stochastic systems <cit.>, used to compute the domain of attraction <cit.>, and employed for control <cit.>.

The Koopman operator has also been used to study nonlinear stability. 
Certain sets of points in phase space that exhibit the same asymptotic behavior are referred to as isostables and play an important role in characterizing transient behavior.
Isostables correspond to level sets of a particular Koopman eigenfunction and can be computed in the entire basin of attraction using, e.g., Laplace averages <cit.>.
Building on <cit.>, it has been further shown <cit.> that special Lyapunov functions from classical stability theory can be determined from Koopman eigenfunctions and a sufficient condition for global stability is the existence of stable eigenfunctions. 
As with the Lyapunov measure, it can be shown that for a stable fixed point there exists 
a special, nonnegative observable V() that satisfies:

    L_U  [V]()<0  ∀≠^*.

Thus, the Lyapunov function decays everywhere under action of the adjoint Liouville operator and is related to its properties and those of the Koopman operator family.
In <cit.>, it has been shown that global stability of a fixed point can be established through the existence of a set of C^1 eigenfunctions of the Koopman operator associated with the eigenvalues of the Jacobian of the vector field, and that Koopman eigenfunctions can be used to define a Lyapunov function and contracting metrics. 
A numerical scheme based on a Taylor expansion is proposed to compute stability properties including the domain of attraction <cit.>.
These ideas have been further extended to examine global stability properties of hyperbolic fixed points and limit cycles and also for non-analytical eigenfunctions <cit.>.






























































 §.§ Observability and controllability






Characterizing a system's observability and controllability is crucial for designing sensor-based estimation and control strategies. 
Linear observability and controllability criteria have been extended to characterize nonlinear systems <cit.>, e.g. local observability can be tested by constructing an observability matrix using Lie derivatives.
Balanced model reduction has also been extended to nonlinear systems <cit.>. 

Operator-theoretic approaches provide new opportunities for determining observability and controllability properties of high-dimensional nonlinear systems from data using linear techniques.



There has been considerable work on nonlinear observability analysis based on the Perron-Frobenius operator formulation for the Lyapunov measure equation. 
Analogously to the observability Gramian for linear systems, which can be obtained as the solution to a Lyapunov equation, the nonlinear equivalent can be obtained from the Lyapunov measure equation. 
In <cit.>, set-oriented methods have been used to compute the observability Gramian for nonlinear systems with output measurements, based on partitioning the phase space into slow and fast time regimes using the Lyapunov measure.  
The linear observability Gramian appears here as a special case of the transfer operator based approach. 
The degree of observability of a set can further be related to the residence time. 
These ideas have been extended to determine sensor and actuator placement for the controlled PDE associated with the infinitesimal generator of the transfer operator <cit.>.
In particular, the observability and controllability is characterized in terms of the support of the observability and controllability matrices, respectively, and the finite-time and infinite-time Gramians are formulated in terms of the transfer operators. 
The infinite-time controllability is only well defined when the vector field () is a.e. uniformly stable; thus, an alternative approach to compute the infinite-time controllability set is based on an exponential discounting of the density, which does not rely on the stability property <cit.>.  
The terms coarse observability and coarse controllability have been introduced to characterize the weaker notion obtained from the finite-dimensional approximation <cit.>.

The Koopman operator perspective may also be used to assess nonlinear observability and controllability.
Generally, two states are indistinguishable if their future output sequences are identical. Thus, a nonlinear system is considered nonlinearly observable if any pair of states are distinguishable <cit.>.
In <cit.>, nonlinear observability is evaluated in the proposed Koopman-based linear observer design framework, which provides a linear representation of the underlying system in terms of the Koopman eigenfunctions (see Sec. <ref>). In particular, the nonlinear system is nonlinearly observable if the pair (,^H) is observable, which can be determined via the rank condition of the corresponding observability matrix. 
These ideas have been applied to study pedestrian crowd flow <cit.>, extended further to input-output nonlinear systems <cit.> resulting in bilinear or Lipschitz formulations, and used to compute controllability and reachability <cit.>. 
The observability and controllability Gramians, which are used to examine the degree of observability and controllability, can be computed in the lifted observable space given by a dictionary of functions <cit.>.  
In this case, the observability/controllability of the underlying system is related to the observability/controllability of the observables.
The underlying assumption is that the state, input, and output are representable in terms of a few Koopman eigenfunctions, which relies on a suitable choice of observable functions to retain the relevant information.





















































 §.§ Observer synthesis for state estimation 

Observer design is critical for sensor-based estimation of high-dimensional systems.  
The observer is a dynamical system that estimates the current state _k from the history of output measurements of the system. 
The Extended Kalman Filter (EKF) is perhaps the most widely used nonlinear observer, benefiting from high performance and a simple formulation based on linearization of the nonlinear state equations. 
However, EKFs are sensitive to noise, do not have convergence guarantees, have limited performance for strongly nonlinear systems, and assume Gaussian noise and disturbances  <cit.>. 

Observer design in the Koopman operator framework involves constructing a state estimator for the system

_k+1 =  (_k, _k),  _k= (_k)

using a nonlinear transformation of states  and outputs  (and inputs for systems with actuation).  This results in the following estimator dynamical system

    _k+1 = _k,  _k 		= ^y(_k),  _k           = ^x(_k),









where ^x and ^H are formed from the Koopman modes and observer Koopman modes.
System (<ref>) can be obtained from a spectral decomposition of the Koopman operator, where  represent Koopman eigenfunction coordinates, which has been referred to as Koopman observer form <cit.> (see also Sec. <ref>).
The resulting system requires that the state and output lie in the span of a subset of eigenfunctions. 
Luenberger or Kalman observers can then be synthesized based on (<ref>), where convergence can be shown under the observability condition based on the Koopman spectral properties, and shown to exhibit increased performance over the extended Kalman filter <cit.>.
This work has been extended to input-output systems yielding Lipschitz or bilinear observers, resulting from the representation of the unforced system in eigenfunction coordinates <cit.>, and for constrained state estimation based on a moving horizon analogous to model predictive control <cit.>.
Building on kernel DMD and the Koopman observer form, a Luenberger Koopman observer has been applied to pedestrian crowd flow to estimate the full state from limited measurements <cit.>.

The probabilistic perspective based on the Perron-Frobenius operator allows one to incorporate uncertainty into observer synthesis. 
Particle filters based on Monte Carlo sampling are commonly used for forecasting and estimation under uncertainty by approximating the PDF with a finite number of samples. 
However, this approach may suffer from the curse of dimensionality <cit.>, particle degeneracy <cit.>, and loss of diversity <cit.>. 
A particular challenge is the accurate estimation of the filter weights.
The Perron-Frobenius and Liouville operator-based approaches have been shown to be advantageous over Monte-Carlo methods <cit.>.
More recently <cit.>, a Bayesian formalism involving the Perron-Frobenius operator has been proposed,

which is shown to achieve superior performance over a generic particle filter and bootstrap filter.
This work has been extended to stochastic systems using a Karhunen-Loeve decomposition of the process noise for uncertainty quantification and state estimation <cit.>.

























































 §.§ Control design

Linear models that capture the dynamics of a nonlinear system provide tremendous opportunities for model-based control, making linear techniques readily applicable to nonlinear systems.
In particular, Koopman-based frameworks are amenable to the application of standard linear control theory, in contrast to the probabilistic formulations based on the Perron-Frobenius operator, which require special care.
Koopman models (see Sec. <ref>) have been increasingly used in combination with optimal control for trajectory steering, e.g. with LQR <cit.>, SDRE <cit.>, and MPC <cit.>.
Besides theoretical developments, Koopman-based MPC has been increasingly applied in realistic problems, such as power grids <cit.>, high-dimensional fluid flows <cit.>, and experimental robotic systems <cit.>. 
Although the predictive power of Koopman models may be sensitive to the particular choice of observables and the training data, MPC provides a robust control framework that systematically compensates for model uncertainty by taking into account new measurements (see also Sec. <ref>).
Optimal control formulations have also been considered for switching problems <cit.>.
Based on a global bilinearization, the underlying dynamical system can be stabilized using feedback linearization <cit.>.
A different strategy aims to shape Koopman eigenfunctions directly, referred to as eigenstructure assignment, which has been examined for fluid flow problems <cit.>.
In contrast to pole placement, which aims to design the eigenvalues of the closed-loop system, eigenstructure assignment aims to additionally modify the structure of the eigenfunctions in a desirable way, which requires additional inputs.
More recently, feedback stabilization for nonlinear systems has been achieved via a control Lyapunov function-based approach based on the bilinear Koopman system in eigenfunction coordinates <cit.>. 
In particular, a convex optimization problem is formulated to search for the control Lyapunov function for the bilinear Koopman system <cit.>.

In many applications it is of interest to shape the distribution of some quantity, e.g. swarms of UAVs or satellites, spin dynamics in nuclear magnetic resonance (NMR) spectroscopy and imaging <cit.>, dispersions in process industries <cit.>, oil spill in the ocean,  search mission design <cit.>, and beam dynamics <cit.>. 
Of particular interest is the shaping of the asymptotic pdf <cit.>, so that the stationary pdf of the closed-loop system coincides with a desired distribution. 
This relies on the idea that temporal averages can be replaced by spatial analogues under Birkoff's ergodic theorem <cit.>, after which the mean or variance of a quantity can be controlled by modifying the stationary pdf of the controlled system. 
Optimal control approaches have also been proposed to modify the pdf in the presence of uncertainty in initial conditions or parameters <cit.>. 


The process of driving one distribution to another one is further intimately related to Monge-Kantorovich optimal transport theory <cit.>.
In <cit.>, optimal transport theory has been used to solve an finite-horizon control problem to achieve a desired distribution, where the optimal control vector field is estimated by solving the associated Liouville equation over the finite horizon. Set-oriented and graph-based methods have also been used to study controllability and optimal transport <cit.>.

Set-oriented representations of the dynamics (see Sec. <ref>) are amenable to optimal control approaches based on dynamic programming <cit.> for discrete-time, discrete-state, and discrete-action problems. The discretized control problem can be interpreted as a Markov decision problem (MDP).
The optimization problem for the MDP can then be posed as a linear program, which has been demonstrated for communication networks <cit.>.
An optimal control formulation in terms of an MDP has been formulated for the discretization Liouville equation and demonstrated for navigation in a maze <cit.>.
Similar ideas have been used for controlling communication networks <cit.> and optimizing the asymptotic pdf over the velocity field in high-dimensional fluid flows <cit.>.
The inverse Frobenius-Perron problem (IFPP) <cit.> formulates the control problem for the stationary pdf as a perturbation problem, which aims to find a perturbed transition matrix and the associated perturbed dynamics that give rise to the target density. 
The Markov transition matrix approximating the Perron-Frobenius operator can also be interpreted as a graph, where the nodes correspond the symbols associated with the discrete states and the vertex weights are the transition probabilities facilitating the application of graph-theoretic methods for optimal or minimal-path searches, e.g. in the context of IFPP <cit.>. 
Stabilization problems have been successfully solved via the Lyapunov measure equation <cit.>, where the problem can be formulated as a linear program, extended for stochastic systems <cit.>, and applied to high-dimensional fluid flows <cit.>. 
A different perspective is assumed in <cit.>, where a convex optimization problem is formulated to determine local perturbations in the context of optimal mixing. 
In particular, the stochastic kernel is designed as a perturbation to the deterministic kernel of the Perron-Frobenius operator, which controls the diffusion and thus the mixing properties of the underlying system. A similar problem has been studied based on the infinitesimal generator <cit.>, but resulting in a convex quadratic program.
More recent work <cit.> considers the perturbation of the stochastic part of the dynamical system and provides a solution in closed form. 

The success of MPC has also driven its use for controlling probability distributions. 
A nonlinear MPC problem using set-oriented representations <cit.> suffers, however, from the curse of dimensionality. 
Thus, the work has been extended using Monte-Carlo and particle filter methods to estimate the pdf, which is referred to as particle MPC <cit.>. In particle MPC the Liouville equation enters the MPC formulation as a constraint to control the mean and variance of the state. 
It is worth noting that data-driven PDF prediction using neural nets is increasingly explored due to faster evaluation/prediction of the PDF in contrast to numerically integrating sample points. The loss metric, i.e. the error between the true and estimated PDF from NN, can be computed using automatic differentiation of neural nets <cit.>, building on physics informed neural net via PDE constraints <cit.>, e.g. the Liouville equation.






































	















	

	


 §.§ Sensor and actuator placement

Determining suitable locations of sensors and actuators for data collection and decision-making is a crucial task in any real-world application. Generally, sensor and actuator placement may refer to selecting spatial locations or specific variables, which are used synonymously here. 
Operator-theoretic placement provides a significant opportunity, as they capture global properties, such as meta-stable states, associated with persistent dynamics, and they generalize to nonlinear systems, e.g. for estimating nonlinear observability and controllability. 
Selecting optimal sensor and actuator locations amounts to a combinatorially hard problem. 
Compressed sensing <cit.>, sparsity-promoting algorithms employing an l_1-penalty term, and sparse sampling techniques such as gappy POD <cit.> have played an increasingly important role in the context of sensor/actuator selection <cit.>. 

In particular, the underlying low-rank structure of the data can be exploited for efficient sensing. 
Thus, compressed sensing and sparsity-promoting algorithms have also been increasingly combined with POD <cit.> and DMD <cit.>. 
Cluster-based reduced-order models (CROM) <cit.>, which provide a coarse approximation of the Perron-Frobenius operator for PDEs, have been combined with the sparse sensor optimization for classification (SSPOC) <cit.> algorithm and pivot locations from the QR factorization (see e.g. discrete empirical interpolation methods (DEIM) <cit.>) applied to the dominant POD modes to learn optimized sensor locations tailored to a specific model <cit.>. Probabilistic dynamics are then preserved if the resulting sensing matrix satisfies the restricted isometry property from compressed sensing.
Recently <cit.>, sensor placement has been explored for multi-scale systems using multi-resolution DMD and DEIM, where sensor locations are tailored to intermittent or transient mode activity.
The control oriented framework DMD with control <cit.> has been unified with compressive DMD <cit.> for compressive system identification <cit.> to identify low-order models from limited input-output data and enable reconstruction of the high-dimensional, interpretable state. This framework, and similarly eDMDc, provide opportunities for optimized placement of sensors and actuators using linear systems techniques. 
A greedy submodular approach was proposed in <cit.> based on the Koopman observer form and a mutual information or observability based criterion. 


Sensor placement and actuator placement can also be performed by leveraging the generator PDEs of the Perron-Frobenius and Koopman families.
Controllability and observability Gramians can be generalized for nonlinear systems based on the (controlled) Liouville and adjoint Liouville equation and subsequently used for sensor and actuator placement by maximizing the support (or the l_2 norm) of the finite-time Gramians <cit.>.  
The approach utilizes set-oriented methods, where sensors/actuators are placed in certain cells, and the location can be optimized by solving a convex optimization problem <cit.>. 
A greedy heuristic approach based on these ideas using set-oriented methods is proposed in <cit.>, which further investigates different criteria such as maximizing the sensing volume (sensor coverage), response time and accuracy (relative measure transported to the sensor in finite time) and incorporating spatial constraints.


The framework has been further extended to incorporate uncertainty <cit.>. 
Balanced truncation has recently been used for efficient placement of sensors and actuators to simultaneously maximize the controllability and observability Gramians <cit.>, which may be promising for Koopman-based Gramians. 








































































§ CONCLUSIONS & FUTURE DIRECTIONS



In this chapter, we have explored a number of applications of the Koopman and Perron Frobenius operators for the control of nonlinear systems. 
Although operator-theoretic analysis has a long history in dynamical systems and control, there has been considerable renewed interest in the past decade.  
Much of this recent progress has been driven by the increased availability of data, along with advanced machine learning and optimization algorithms.  
These data-driven approaches are providing approximations to the Koopman and Perron Frobenius operators that are then used for control. 
We have introduced a general control formulation and discussed how this may benefit from a coordinate transformation where nonlinear dynamics become approximately linear.  
Considerations such as stability, controllability and observability, controller and observer design, and sensor/actuator placement were discussed in the context of operator-theoretic approaches, highlighting recent theoretical and applied advances. 
We also discussed a number of important embedding strategies, such as extended DMD, delay coordinates, and eigenfunctions, which may all be used for these control objectives.  
Finally, we provided an example that leverages these various embeddings for model predictive control, showing that MPC is remarkably robust to model uncertainty.  

Despite the tremendous recent progress in operator-based control, there are a number of limitations that must be addressed in the future.  
Obtaining accurate representations of the Koopman operator from data is a key enabler of model-based control. 
However, identifying a coordinate system that is closed under the Koopman operator is notoriously difficult, as the eigenfunctions may have arbitrarily complex representations.  
Fortunately, emerging techniques in deep learning are providing powerful representations for these linearizing transformations <cit.>. 
Another key challenge is the existence of a continuous eigenvalue spectrum in chaotic dynamical systems, which complicates matters considerably.  
Recent methods in delay coordinates <cit.> and tailored neural network architectures <cit.> are making headway, although this is far from resolved.  
However, at this point, these advanced embedding techniques have not been adapted for control design, which is an exciting avenue of ongoing work.  

Although much of the focus in data-driven Koopman and Perron Frobenius theory has been on obtaining increasingly accurate approximate embeddings, it may be that improved models have marginal benefit in control design.  
Certainly improved models are useful for prediction and estimation, but advanced control techniques, such as model predictive control, are remarkably robust to lackluster models.  
In many cases, it may be that a simple DMDc model may work nearly as well as a sophisticated model based on the lates embedding techniques.  
However, future operator-based controllers will need to be certified, requiring guarantees on the model fidelity and a rigorous quantification of errors and uncertainties.  
Importantly, the model uncertainty is intimately related to the quality and quantity of the training data, the richness of the dynamics sampled, and the basis used for regression.  
Future efforts will undoubtedly continue to incorporate known or partially known dynamics, structure, and constraints in the data-driven regressions, improving both the models and the uncertainty bounds.  

The current state of the art in operator-theoretic control, with a compelling set of success stories and a long list of future work, make it likely that this field will continue to grow and develop for decades to come.  
It is inevitable that these efforts will increasingly intersect with the fields of machine learning, optimization, and control.  
Advances in machine learning and sparse optimization will continue to drive innovations, both for model discovery and for sensor placement, which are closely related to Koopman and Perron Frobenius.  
Likewise, concrete successes in the control of nonlinear systems will continue to motivate these advanced operator-based approaches.  




















































	EK gratefully acknowledges support by the “Washington Research Foundation Fund for Innovation in Data-Intensive Discovery" and a Data Science Environments project award from the Gordon and Betty Moore Foundation (Award #2013-10-29) and the Alfred P. Sloan Foundation (Award #3835) to the University of Washington eScience Institute, and funding through the Mistletoe Foundation.
	SLB and JNK acknowledge support from the Defense Advanced Research Projects Agency (DARPA contract  PA-18-01-FP-125). SLB acknowledges support from the Army Research Office (W911NF-17-1-0306 and W911NF-17-1-0422). JNK acknowledges support from the Air Force Office of Scientific Research (FA9550-19-1-0011).














plain        
 
