ient Shared Control {#sec:NGSC}
===============================

Expressing Shared Control as an Optimization Problem
----------------------------------------------------

Let $s \in \mathcal{S}$ be the state of the system. Let $a^H \in \mathcal{A^H}$ as the user action, $a^R \in \mathcal{A^R}$ be the autonomous robot action, and $u \in \mathcal{U}$ be the shared control action. The human and the robot agent each select actions following their stochastic policies, $\pi_H$, and $\pi_R$. Our goal is to find a shared control policy $\pi_S$ that solves the following optimization problem.

[u\_t]{}[Q\_H (s\_t, u\_t)]{} \[opt:problem\]

The shared control policy is chosen to maximize the user’s internal action-value function $Q_H(s_t, u_t)$ at each step. Although it is possible to learn $Q_H$ using methods such as MaxEnt IOC [@ziebart2009planning], predicting the user controls $a^H_t$ can be challenging due to interpersonal differences. Instead, we regard the user action as an estimate of $\nabla_s Q_H(s_t, a^H_t)$ at each step.

The constraint on the KL-divergence between the robot policy and the shared policy ensures that the shared policy does not deviate far from the autonomous robot policy. The problem can be expressed using a Lagrange Multiplier, assuming a linear approximation of our objective $\nabla_s Q_H(s_t, a^H_t)$ and a quadratic approximation of the KL-divergence constraint. Solving this approximation of the Lagrangian leads to an update rule which introduces natural gradient adaptation.

Natural Gradient Shared Control {#natural-gradient-shared-control}
-------------------------------

We introduce a new control paradigm for shared control using natural gradient adaptation. Note that our goal is to find the action that maximizes $Q_H$, resulting in taking gradient steps in the direction of ascent. $$\label{eqn:state_update}
    \begin{aligned}
    s_{t+1} &= s_t + \eta F(s_t)^{-1} \nabla_s Q_H(s_t, a^H_t)  \\
    &=s
ient Shared Control { # sec: NGSC } 
 = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 

 Expressing Shared Control as an Optimization Problem 
 ---------------------------------------------------- 

 Let $ s \in \mathcal{S}$ be the state of the arrangement. lease $ a^H \in \mathcal{A^H}$ as the user action, $ a^R \in \mathcal{A^R}$ be the autonomous robot natural process, and $ u \in \mathcal{U}$ be the share command action. The homo and the automaton agent each select military action following their stochastic policies, $ \pi_H$, and $ \pi_R$. Our goal is to find a shared control policy $ \pi_S$ that solve the following optimization problem. 

 [ u\_t]{}[Q\_H (s\_t, u\_t) ] { } \[opt: problem\ ] 

 The shared command policy is chosen to maximize the user ’s internal action - value affair $ Q_H(s_t, u_t)$ at each step. Although it is possible to learn $ Q_H$ using methods such as MaxEnt IOC   [ @ziebart2009planning ], predict the user controls $ a^H_t$ can be challenging due to interpersonal differences. Instead, we regard the user action as an estimate of $ \nabla_s Q_H(s_t, a^H_t)$ at each step. 

 The constraint on the KL - divergence between the robot policy and the shared policy ensures that the shared policy does not deviate far from the autonomous robot policy. The trouble can be express using a Lagrange Multiplier, assume a analogue estimate of our objective $ \nabla_s Q_H(s_t, a^H_t)$ and a quadratic approximation of the KL - divergence constraint. Solving this approximation of the Lagrangian leads to an update rule which introduce natural gradient adaptation. 

 Natural Gradient Shared Control { # natural - gradient - shared - control } 
 ------------------------------- 

 We introduce a new dominance paradigm for shared control using lifelike gradient adaptation. Note that our finish is to find the action that maximizes $ Q_H$, resulting in taking gradient steps in the focus of ascent. $ $ \label{eqn: state_update } 
     \begin{aligned } 
     s_{t+1 } & = s_t + \eta F(s_t)^{-1 } \nabla_s Q_H(s_t, a^H_t)   \\ 
     & = s
ienh Shared Control {#sec:NGSC}
===============================

Txpressing Shared Contron as ah Optimixation Problem
----------------------------------------------------

Let $s \in \mathcel{S}$ ve tht state of the systdm. Let $a^H \in \mathxal{A^I}$ as the user acvjon, $a^R \lu \matggal{A^R}$ ue the autonomoos robot acthon, and $u \in \mdtfccl{U}$ be the shared control action. The human snf the robot aggnt essh sslect actions following their stocgastic kolicies, $\pi_H$, and $\po_R$. Our goal is to find a sjaref control policy $\pl_S$ that solces eye following optimizatpmn problem.

[o\_t]{}[Q\_H (s\_t, u\_t)]{} \[opt:problem\]

The shared zontrpl policy us chlven to maxinize nhe user’s intcgnal acdion-valie function $Q_H(x_t, n_t)$ ar each step. Although mt is possible to lewrn $Q_H$ ushny methods such as MaxWnr IOC [@eiebast2009plxbnivg], krevicfing tje nser controms $a^H_t$ can ve challenging due uo ybterpersonal sifferqnses. Instead, we regard the user action av ah estimate of $\nabla_s Q_H(w_t, a^H_t)$ at each step.

Thg constraigt on the KL-divergence between the robot policy atd thx rhaxcq popicy ensures that the shared policy does not qsvoane far from the amtonomous robot pokify. Jhe problem cav be erlrsssed using a Lagrwnge Mujtipluer, assumyng s linear approximation of oyr objective $\babla_s Q_H(s_t, a^H_t)$ anb a quadratie apprpximayion of the KL-divergencz consfraint. Solvlng this zoproximation of ghe Ldgrangian leads to an updaee rule wiich nntroducds nstural gradient wdaptation.

Natural Gradiejt Shcred Wontrol {#nahural-gradient-shared-control}
-------------------------------

We invcoduce a new vottrml paradngm fov shared controj using naturak gradiznt adxptation. Nkte thav our goal if to find the wction that kaximizef $Q_H$, resylting kv taking gradirnt steps in the dieection of ascent. $$\kabgl{sqn:state_update}
    \vegin{aligned}
    s_{t+1} &= s_e + \eva F(s_e)^{-1} \nabla_s Q_H(s_t, a^H_t)  \\
    &=s
ient Shared Control {#sec:NGSC} =============================== Expressing Shared an Problem ---------------------------------------------------- $s \in \mathcal{S}$ system. $a^H \in \mathcal{A^H}$ the user action, \in \mathcal{A^R}$ be the autonomous robot and $u \in \mathcal{U}$ be the shared control action. The human and the agent each select actions following their stochastic policies, $\pi_H$, and $\pi_R$. Our goal to a control $\pi_S$ that solves the following optimization problem. [u\_t]{}[Q\_H (s\_t, u\_t)]{} \[opt:problem\] The shared control policy is to maximize the user’s internal action-value function $Q_H(s_t, at each step. Although is possible to learn $Q_H$ methods as MaxEnt [@ziebart2009planning], the controls $a^H_t$ can challenging due to interpersonal differences. Instead, we regard the user action as an estimate of $\nabla_s Q_H(s_t, at each constraint on KL-divergence the policy and the ensures that the shared policy does from the autonomous robot policy. The problem can expressed using Lagrange Multiplier, assuming a linear approximation our objective $\nabla_s Q_H(s_t, a^H_t)$ and a quadratic of the KL-divergence constraint. Solving this approximation of the Lagrangian leads to an update rule natural gradient adaptation. Natural Shared Control {#natural-gradient-shared-control} We a control for shared using natural gradient adaptation. Note that our goal is to find action that maximizes $Q_H$, resulting in taking gradient steps in of $$\label{eqn:state_update} \begin{aligned} s_{t+1} s_t + \eta F(s_t)^{-1} Q_H(s_t, \\ &=s
ient Shared Control {#sec:NGSC}
===============================

EXpressing SHared conTroL aS an OPtimIzation Problem
----------------------------------------------------

lEt $s \iN \mathcal{S}$ be the state of tHe sysTeM. let $a^h \In \MathcAl{A^H}$ as tHE uSER acTiOn, $A^R \iN \mAThCal{A^R}$ Be tHe autonOmous robot ActIoN, and $u \in \mathcAL{U}$ Be the shareD coNtrol action. THe hUman anD tHe rOBot agEnt Each sElect aCTions fOllowing tHeIR stochAStic polICIeS, $\pi_H$, And $\pi_R$. Our goal is to FInD A shared control Policy $\Pi_s$ ThAT SolVes The followiNg OptimIZation pRObLEM.

[U\_t]{}[Q\_h (S\_t, u\_t)]{} \[opt:probleM\]

The shared cONtrOl poliCy Is cHOsen to MaximIzE The User’s internAl acTion-value FunctiON $Q_H(s_t, u_t)$ AT each stEp. AlthOugH it Is poSSiBlE to LeARn $Q_h$ UsIng MEthOds such aS MAxent IOc [@zieBART2009PlanNinG], preDictiNg the user contRolS $a^H_t$ CAn bE chalLengiNg duE tO inteRpersoNal diFfErences. Instead, wE regArd the useR acTiOn aS aN estiMAte of $\nAblA_s Q_h(s_t, a^H_t)$ aT each stEP.

ThE cONSTrAint on the KL-divergeNcE BEtWeen the rObot poLIcY aND the sharEd PolIcy eNSUres tHat tHE sHared polIcy doeS NoT dEviate fAr From thE aUtoNomOus roBOt poLicy. ThE problem Can be EXpressed using a lAgrange MultipLIeR, ASsUMing A liNear approxiMatiON of oUr obJEcTivE $\Nabla_S Q_H(s_t, A^H_T)$ AnD A quadratic approximaTiOn of thE KL-diVergence constRaint. SolviNG THis approXimaTIoN Of the LagrangiaN leadS to an updatE Rule whicH intrOduces naTural gradIENt adaptaTioN.

NaTurAl GRADiEnt Shared ContROL {#natUrAl-gradiEnt-Shared-cOntRol}
-------------------------------

we iNtrOdUce a new coNtrol parAdIgM fOr ShaRed coNTrol usinG nAtuRaL grAdienT AdaptaTion. NOte tHaT oUR goAl is to fINd THE actIoN tHat mAxiMiZes $Q_H$, ResuLTinG in takiNg gradienT stEPs in ThE dIrectioN of ascent. $$\labeL{eQn:state_updAtE}
    \beGin{aliGNEd}
    s_{t+1} &= s_t + \etA F(s_t)^{-1} \nabla_s Q_H(s_t, a^H_t)  \\
    &=s
ient Shared Control {#sec: NGSC}
==== ===== === === == ==== ==== ======

Expres s ingShared Control as an O ptimi za t ionP ro blem------- - -- - - --- -- -- --- -- - -- ----- --- ------- ---------- 

L et  $s \in \mat h ca l{S}$ be t hestate of the  sy stem.Le t $ a ^H \i n \ mathc al{A^H } $ as t he user a ct i on, $a ^ R \in \ m a th cal{ A^R}$ be the auto n om o us robot actio n, and  $ u  \ i n  \m ath cal{U}$ be  t he sh a red con t ro l a cti o n. The humanand the rob o t a gent e ac h s e lect a ction sf oll owing their  sto chastic p olicie s , $\pi_ H $, and$\pi_R $.Our  goa l  i stofi n d a sh are d  co ntrol po li cy  $\pi _S$t h a t  sol ves  the  foll owing optimiz ati on p r obl em.
[u\_t ]{}[ Q\ _H (s \_t, u \_t)] {}  \[opt:problem\ ]

T he shared  co nt rol  p olicy is cho sen  to  maximi ze theu ser ’s i n te rnal action-valuefu n c ti on $Q_H( s_t, u _ t) $a t each s te p.Alth o u gh it  isp os sible to  learn $Q _H $ using  m ethods  s uch  as  MaxE n t IO C [@zi ebart200 9plan n ing], predicti n g the user co n tr o l s$ a^H_ t$can be chal leng i ng d ue t o  i nte r perso nal d if f er e nces. Instead, we r eg ard th e use r action as a n estimate o f  $\nabla _s Q _ H( s _t, a^H_t)$ at  each  step.

Th e  constra int o n the KL -divergen c e  between  th e r obo t p o l ic y and the sha r e d po li cy ensu res  that t hesha red  po li cy does n ot devia te  f ar  f rom  thea utonomou srob ot  po licy. The pr oblem  can  b ee xpr essed u s in g a La gr an ge M ult ip lier,  ass u min g a lin ear appro xim a tion  o four obj ective $\nabl a_ s Q_H(s_t,  a ^H_ t)$ an d a quadra tic approximation of th e  KL-div erg encecons traint. S olv ing th isa pproxi mation  of t he  La g r angia n le ads  t o an updat e rul e whi ch  int roduces  natural gradienta dap tation.

Natu ral  Gra d i en t S h ar e d C on t rol { #natural-gradie nt-shared- co n tr ol}
------ - --- -- ------- ------- ----- 
We int roduce anew contr ol  par a d igm  for share d contro l using n a tural gr adien t a daptat io n.Notethat o u r g oal i s to f in d theactio nthat max imizes $Q_H$, resulting  in ta kinggra dient ste psi n t he direct ionof ascent.  $$ \la bel{e qn: s tate_ upda t e} 
  \beg in{a l igned}
  s _{t + 1 }&= s_t + \e t a F(s _t)^{ -1} \nabla _s Q _H(s_t, a^H_t)  \ \ 
    &=s
ient Shared_Control {#sec:NGSC}
===============================

Expressing_Shared Control as an_Optimization Problem
----------------------------------------------------

Let_$s_\in \mathcal{S}$_be_the state of_the system. Let_$a^H \in \mathcal{A^H}$ as_the user action,_$a^R_\in \mathcal{A^R}$ be the autonomous robot action, and $u \in \mathcal{U}$ be the shared_control_action. The_human_and_the robot agent each select_actions following their stochastic policies,_$\pi_H$, and_$\pi_R$. Our goal is to find a shared_control_policy $\pi_S$ that_solves the following optimization problem.

[u\_t]{}[Q\_H (s\_t, u\_t)]{} \[opt:problem\]

The shared_control policy is chosen to maximize_the user’s internal_action-value_function_$Q_H(s_t, u_t)$ at each_step. Although it is possible to_learn $Q_H$ using methods such as_MaxEnt IOC [@ziebart2009planning], predicting the user controls $a^H_t$_can be challenging due to interpersonal_differences. Instead, we regard the_user action_as an estimate of $\nabla_s_Q_H(s_t, a^H_t)$ at_each step.

The_constraint on the_KL-divergence between the robot policy and_the shared policy_ensures that the shared policy does_not_deviate far from_the_autonomous_robot policy._The problem can_be_expressed using_a_Lagrange Multiplier, assuming a linear approximation_of_our objective $\nabla_s Q_H(s_t, a^H_t)$ and a_quadratic approximation of the_KL-divergence_constraint. Solving this approximation_of the Lagrangian leads to_an update rule which introduces natural_gradient adaptation.

Natural_Gradient Shared_Control {#natural-gradient-shared-control}
-------------------------------

We introduce a new control paradigm for shared control using_natural gradient adaptation. Note that our_goal is to find_the action_that_maximizes $Q_H$, resulting_in_taking gradient_steps in the direction of ascent. $$\label{eqn:state_update}
_ _ \begin{aligned}
    s_{t+1}_&= s_t + \eta_F(s_t)^{-1}_\nabla_s Q_H(s_t, a^H_t)  \\
 _  &=s