Extreme Spin Squeezing from Deep Reinforcement Learning
Feng Chen Jun-Jie Chen Ling-Na Wu Yong-Chun Liu Li You


























State Key Laboratory of Low-Dimensional Quantum Physics, Department of Physics, Tsinghua University, Beijing 100084, China






State Key Laboratory of Low-Dimensional Quantum Physics, Department of Physics, Tsinghua University, Beijing 100084, China



lnwu@pks.mpg.de
Max Planck Institute for the Physics of Complex Systems, Nöthnitzer Straße 38, 01187 Dresden, Germany


ycliu@tsinghua.edu.cn
State Key Laboratory of Low-Dimensional Quantum Physics, Department of Physics, Tsinghua University, Beijing 100084, China
Frontier Science Center for Quantum Information, Beijing, China
Collaborative Innovation Center of Quantum Matter, Beijing 100084, China


lyou@tsinghua.edu.cn
State Key Laboratory of Low-Dimensional Quantum Physics, Department of Physics, Tsinghua University, Beijing 100084, China
Frontier Science Center for Quantum Information, Beijing, China
Collaborative Innovation Center of Quantum Matter, Beijing 100084, China
Beijing Academy of Quantum Information Sciences, Beijing 100193, China
















Spin squeezing (SS) is a recognized resource for realizing measurement precision beyond the standard quantum limit ∝ 1/√(N).
The rudimentary one-axis twisting (OAT) interaction can facilitate SS and has been realized in diverse experiments, but it cannot achieve extreme SS for precision at Heisenberg limit ∝ 1/N.
Aided by deep reinforcement learning (DRL), we discover size-independent universal rules for realizing nearly extreme SS with OAT interaction using merely a handful of rotation pulses. 
More specifically, only 6 pairs of pulses are required for up to 10^4 particles, while the time taken to reach extreme SS remains on the same order of the optimal OAT squeezing time, which makes our scheme viable for experiments that reported OAT squeezing. 

This study highlights the potential of DRL for controlled quantum dynamics.








Extreme Spin Squeezing from Deep Reinforcement Learning
    Li You
    December 30, 2023
=======================================================








Squeezed spin state <cit.> refers to a class of spin entangled state
whose uncertainty in one spin component perpendicular to the mean spin direction
is smaller than the classical limit √(N)/2 of a coherent spin state (CSS) of N polarized (pseudo-) spin-1/2 particles (ħ =1 hereafter).
It can be applied in quantum metrology, e.g., through Ramsey interferometry <cit.>, to reach precision beyond the standard quantum limit ∝ 1/√(N).
Extreme spin squeezing (SS) with precision at Heisenberg limit (HL) ∝ 1/N is realized from two-axis counter-twisting (TACT) interaction <cit.>, although unfortunately TACT interaction does not arise easily in any known systems. 
Proposals for generating effective TACT interactions <cit.> are faced with challenges in their respective realizations.
An alternative approach to realize SS is based on one-axis twisting (OAT) interaction <cit.>, which occurs ubiquitously in systems of interacting spins, e.g., through large-detuned atom-photon coupling in an optical cavity <cit.> or via atomic collisions in a Bose-Einstein condensate <cit.>.
However, SS from OAT interaction only scales as 1/N^2/3 <cit.>, which falls short of the HL of extreme squeezing ∝ 1/N. 
Hence, developing approaches capable of improved SS based on OAT interaction constitutes a topical area of research.

A prominent model that has attracted widespread attention is to augment OAT interaction with a transverse coherent field <cit.>.
Promising proposals based on this model include, for instance, transforming OAT to TACT by using a periodic train of ±π/2 spin rotation pulses <cit.>, or a periodically modulated transverse field <cit.>.
Their applications however call for rather stringent experimental conditions, with the former requiring a large number of high-precision pulses, while the latter demanding a high modulation frequency <cit.>.
An alternative proposal makes use of only a few optimized pulses along the mean spin direction <cit.>. 
While improvement over OAT is achieved, the evolution time required is greatly prolonged. 
It makes this scheme less robust against particle losses or decoherence. 
Despite intense efforts devoted over the years, it remains a difficult and unsolved task to find an experimentally feasible scheme implementable with state-of-the-art lab techniques.


One promising solution is to seek help from deep reinforcement learning (DRL), which has attracted much attention in recent years for its effective and flexible handling of dynamical systems. 
Without prior knowledge <cit.>, DRL is capable of achieving an optimized policy and providing perspectives and solutions otherwise impossible or difficult to comprehend in optimal control.
The advantages of DRL over traditional theories and algorithms are already demonstrated with applications in quantum phase transition <cit.>, quantum state preparation <cit.>, and quantum gate control <cit.>, etc. 
However, DRL is computationally demanding and consumes enormous amount of resources for quantum systems with large degrees of freedom. 
For example, a typical DRL process requires thousands or even millions of training episodes, hence is prohibitive to directly handle systems with large particle number N.


In this Rapid Communication, aided by DRL, we propose a scheme applicable in large-sized systems for realizing extreme SS based on OAT interaction with a small number of rotation pulses. 
We first employ DRL in small-sized systems to find out size-independent universal rules for squeezing manipulation.
These rules are then applied to systems with large particle numbers. 
Limited computational resources are found to be sufficient even for quantum systems with large degrees of freedom. 
For N=10^4, we show that extreme SS close to the HL can be reached using only 6 pairs of rotation pulses applied at selected instants. 
Moreover, the evolution time required to reach optimal squeezing is on the same order of that for OAT, implicating that it is potentially robust against particle loss or decoherence.


We begin by describing our scheme applied to the model system of a condensate with pseudo-spin-1/2 atoms, whose Hamiltonian is given by <cit.>

    H = χ J_z^2 + Ω(t)J_y.

Here J_μ = ∑_kσ_μ^(k)/2 (μ=x,y,z) defines collective spin components with the Pauli matrices σ_x,y,z^(k) for the k-th atom.
The Hamiltonian (<ref>) is composed of two terms: the first term χ J_z^2 describes OAT interaction of strength χ, which originates from, for example, spin exchange between atoms <cit.>; the second term Ω(t)J_y denotes the interaction between atoms and the external field with Rabi frequency Ω(t), which can be used to effect a spin rotation of angle θ= ∫^t_0+Δ t_t_0Ω(t)dt along the y-axis over a short pulse duration Δ t.
Starting from a CSS polarized along the x-axis with isotropic distributions for the transverse spin component fluctuations, the bare OAT interaction squeezes the transverse distribution and gives rise to a squeezed spin state with an optimal SS coefficient ∝ 1/N^2/3 <cit.>.
This study aims at reaching extreme squeezing, by applying the external field with controlled pulse sequence.
To make it experimentally feasible, the control pulse manipulations should be few and the evolution time should be short.









First we employ DRL to obtain the pulse sequence for small-sized systems with N≤100. 
The total training duration t_c is set to be around the optimal squeezing time of the effective TACT (with reduced strength χ/3) <cit.>. 
It is divided into t_c/δ t steps with fixed interval δ t.
A typical DRL training is described as follows (more details can be found in the Supplementary Material <cit.>). At each time step t, the agent observes environment (atoms here) through state s_t ∈𝒮 and takes action a_t ∈𝒜 guided by some policy ϖ(a_t|s_t).
The environment then evolves to s_t+1 and returns a scalar reward r_t ∈ℛ back to the agent.
More specifically for our problem, the “state” (𝒮) contains five observables s ∈{J_x,J_y,J_z,J_x^2,J_z^2}.
The “action” (𝒜) consists of three discrete operations a ∈{0,±θ}, where a=0 means evolving for a time interval δ t under the bare OAT interaction, while a=±θ means a short rotation pulse with angle ±θ applied before a δ t evolution under the OAT interaction.
The SS coefficient ξ^2=4(Δ𝐉_)_min^2/N <cit.> is employed as “reward” (ℛ), where (Δ𝐉_)_min^2 denotes the minimal fluctuation of the transverse spin component perpendicular to the mean spin direction.
To avoid sparse rewarding <cit.> and to improve learning efficiency, we decompose the total achievable reward into a logarithm sum

    R_tot=∑_j=1^nr_j=∑_j=1^nlog_10ξ^2(t_j-1)/ξ^2(t_j),

such that at each time step t_j, a reward r_j, the instantaneous decrease of SS coefficient, is fed back to the agent.
Proximal policy optimization (PPO) algorithm <cit.> is employed to update the policy ϖ(a|s) due to its stability and sophisticated control over the training processes.
After training, the PPO algorithm adopts an optimal stochastic policy which returns a distribution in action space ϖ^*(a|s), satisfying ∑_a∈𝒜ϖ^*(a|s) ≡ 1 for a given s.
A specific protocol, i.e., pulse sequence, can be generated from the optimal policy ϖ^*(a|s) in two ways.
One chooses the action (at each time step) a = max_a'ϖ^*(a'|s) with the maximum probability (max-probability for short)
and the other selects the best among multiple protocols generated from a sampling based on ϖ^*(a|s) (with-selection  for short).
In actual DRL training tasks, a neural network is used to parameterize the actor and critic network employed in PPO.



Figure <ref>(a) shows the training result (max-probability) and its comparison with that of OAT and the effective TACT for N = 100.
Clearly, the result from DRL outperforms OAT, and the realized optimal SS approaches (or even outperforms) the effective TACT. 
Only two pairs of pulses are required for reaching the optimal SS, to be respectively applied at χ t≃ 0.045 and χ t≃ 0.068. 
The corresponding time for the optimal SS is χ t≃ 0.075, which is on the same order of the optimal OAT squeezing time (χ t ≃ 0.055).


We next apply the optimal policy trained by DRL for N = 100 to systems with different particle numbers. 
The results are shown in Fig. <ref>. 
The optimal policy is found to perform well for 50 ≤ N ≤ 150. 
However, for larger N ≳ 300, it fails to reach extreme SS (even for selected protocol). 
Therefore, the optimal policy found by DRL is size-dependent, and re-training is typically required for larger systems. 
Nevertheless, DRL process is relatively hard to train, especially for large-sized systems. 
For typical condensates containing 10^3∼ 10^5 atoms or more, besides demanding for a large amount of computational resource, the increasingly larger Hilbert space dimension also makes it prohibitively expensive to explore the complete state space effectively.





Fortunately, from the results trained by DRL for small-sized systems, size-independent universal rules are discovered in the squeezing manipulation. 
First, we note θ and -θ pulses always appear in pairs, with θ=π/2 giving the best performance. 
Other θ also works well and an example of θ=π/3 is included in the Supplementary Material <cit.>. 
Second, the first pulse pair appears at the moment when the effective TACT begins to outperform OAT. 
Third, the subsequent pulse pairs are applied whenever the current squeezing becomes inferior to that of OAT. 
These rules can be understood in physics as constituting a controlled rectification of over-twisting <cit.>. 
The bare OAT interaction twists the distribution of a spin state on the Bloch sphere around the z-axis nonlinearly, which is accompanied by a clockwise nonlinear rotation of the state around the x-axis.

This causes over-twisting of the state at later times, degrading the squeezing performance <cit.>. 
We note the combination of the ±π/2 pulse pair found above can be described by the time evolution operator

    U = e^iπ/2J_ye^-iχ J_z^2δ te^-iπ/2J_y≈ e^-iχ J_x^2δ t,

which corresponds to nothing but a nonlinear rotation operation along the x-axis [see Fig. <ref>(b)].
The ±π/2 pulse pair therefore rectifies the nonlinear over-twisting around the x-axis, such that squeezing can be continuously improved until the state becomes over-twisted again. 
It is worthy to note that the scheme found by DRL is quite different from previous results of enhanced SS with controlled pulses. 
For instance, different from Ref. <cit.>, the pulse sequence found here is nonperiodic, hence the system dynamics cannot be described by an effective TACT Hamiltonian. 
Furthermore, exact ±π/2 pulses are required in Ref. <cit.>, while the rectification of over-twisting here is independent of the specific choice of pulse area. For example, θ=π/3 also works well as shown in the Supplementary Material <cit.>.






In particular, the rectification we find here is nonlinear, which facilitates significantly enhanced SS within a short evolution time. 
This desirable feature is absent when using a transverse field along the mean spin direction <cit.> which only provides a linear complementary rectification. 
In the linear case, either greatly prolonged evolution time is required to obtain enhanced SS <cit.>, or efficient rectification to speed up SS is only maintained for a short time without improving the overall SS level <cit.>.
 







Based on the above summarized universal rules and the insightful understanding gained, we next develop a DRL-inspired scheme (IS for short) applicable to large-sized systems yet demanding only for rather limited computation resource. 
Specifically, for a given N, we calculate the system dynamics and apply pulse sequences following the above three rules, with the duration δ t between the ±π/2 pulses in a pulse pair further optimized by the grid search method <cit.>.


In Fig. <ref>(a) we plot the optimal SS coefficient ξ^2 for systems with varying N. 

For IS, the optimal SS coefficient with respect to N is found to scale as ξ_IS^2 ∝ 1/N^0.96, which is significantly better than the OAT case of ξ_OAT^2 ∝ 1/N^2/3, and approaches the extreme limit ∝ 1/N. 
Moreover, the required number of pulse pairs increases rather slowly as N becomes larger [gray shaded region in Fig. <ref>(a)]. 
For N=10^4, six pairs of pulses are essentially sufficient. 
In addition, the evolution time to achieve optimal SS remains on the same order of that for OAT or the effective TACT [see Fig. <ref>(b)]. 
The inverse power law scaling with respect to N is found to be χ t_IS∝ 1/N^0.6, which is very close to that for OAT with χ t_OAT∝ 1/N^2/3.
We further illustrate the squeezing properties of our IS using an alternative definition of SS coefficient ξ^2_R = N(Δ𝐉_)_ min^2/𝐉^2 with 𝐉 being the mean spin length <cit.>, a coefficient which directly related to the measurement precision as it specifies the noise-to-signal ratio in Ramsey interferometry <cit.>.
From Fig. <ref>(c), we see that the performance of the IS is even better with this squeezing definition. 
The fitted power law scales as ξ_R,IS^2 ∝ 1/N^0.98, and χ t_R,IS∝ 1/N^0.6.


Since only a small number of pulses are required, the IS can be potentially robust against noise or stochastic error.
In actual experiments, the dominant source of error often comes from fluctuations in temporal instants of the pulse pairs or in the duration δ t within each pulse pair.
We perform numerical simulations by modeling these two sorts of errors as Gaussian stochastic processes.
The consequent optimal squeezing coefficient ξ^2 under 1% noise level is shown in Fig. <ref>, which clearly implicates that the IS is rather robust against the aforementioned noises. 
In a previously reported OAT experiment <cit.> involving two hyperfine states |1,1⟩ and |2,-1⟩ of ^87Rb atoms, atomic coupling strength reaches χ = 2π× 0.063 Hz. At N =2000, a 1% uncertainty in time duration amounts to δ t ∼ 0.4 μs, while a 1% uncertainty in the temporal instant of the pulse corresponds to ∼ 0.3 ms.
Both limits are easily satisfied within current experimental capabilities.


Particle loss or detection noise in real experiments also degrade the level of achievable squeezing, as they are known to prevent from reaching the OAT limit <cit.>.
The potential limitations of these factors on our protocol are expected to be on the same level as in reported OAT experiments, due primarily to the comparably required time duration to reach optimal SS.
Further improvements can come from continued development of experimental techniques <cit.>.
Once the OAT limit is achieved, the advantages of our scheme can be readily demonstrated.


In conclusion, we propose to realize extreme SS based on OAT using a few control pulses.
Our scheme is based on size-independent universal rules for squeezing manipulation, discovered with the assistance of DRL.
They can be understood as facilitating nonlinear rectification of over-twisting associated with OAT continuously, hence precisely direct the evolution to optimal SS.
Based on our IS, realizable SS coefficient is found to scale as ∝ 1/N^0.98 significantly surpassing the bare OAT limit of ∝ 1/N^2/3, and approaches the extreme SS limit ∝ 1/N. 
The controlled manipulations required are modest, composing of only a few rotation pulses applied at appropriate instants. For example, 6 pairs of pulses are sufficient for a system with up to 10^4 particles.
Moreover, the time to realize optimal SS is found to remain on the same order of the optimal OAT squeezing time, which indicates that our scheme can be as robust as OAT with respective to particle loss or decoherence.

In our scheme, DRL plays a crucial role in providing insights for discovering universal rules, which paves the way for an experimentally feasible scheme within the state-of-the-art lab techniques.

Our work highlights the great potential of applying DRL to controlled quantum dynamics and quantum state engineering.





We thank Q. Liu for fruitful discussions.
This work is supported by the National Key R&D Program of China (Grant No. 2018YFA0306504)
and the NSFC (Grant No. 91636213, No. 11654001, and No. 91736311).

Y.C. Liu acknowledges support from the NSFC (Grant No. 91736106,  11674390, 91836302).
L. You also acknowledges support from BAQIS (Grant No. 20191550064).









37
fxundefined [1]
 ifx#1

fnum [1]
 #1firstoftwo
 secondoftwo
 
fx [1]
 #1firstoftwo
 secondoftwo
 





noop [0]secondoftwo

ref[1]@startlink#1@href
href[1]#1@endlink
anitize@url [0]`
12`$12`&12`#12`1̂2`_12`%12
startlink[1]
endlink[0]

rl [1]href #1












@bib@innerbibempty

[Kitagawa and Ueda(1993)]ueda1993
  author author M. Kitagawa and author M. Ueda, 10.1103/PhysRevA.47.5138 journal
  journal Phys. Rev. A volume 47,  pages 5138 (year 1993)NoStop
[Wineland et al.(1992)Wineland, Bollinger, Itano, Moore, and Heinzen]ramsey1
  author author D. J. Wineland, author J. J. Bollinger, author W. M. Itano, author F. L. Moore,   and author D. J. Heinzen,  10.1103/PhysRevA.46.R6797 journal journal Phys. Rev. A volume 46, pages R6797 (year 1992)NoStop
[Wineland et al.(1994)Wineland, Bollinger, Itano, and  Heinzen]ramsey1_2
  author author D. J. Wineland, author J. J. Bollinger, author W. M. Itano,  and author D. J. Heinzen, 10.1103/PhysRevA.50.67 journal
  journal Phys. Rev. A volume 50,  pages 67 (year 1994)NoStop
[Ma et al.(2011)Ma,
  Wang, Sun, and Nori]ma2011
  author author J. Ma, author X. Wang, author C. Sun,  and author
  F. Nori, https://doi.org/10.1016/j.physrep.2011.08.003 journal
  journal Physics Reports volume 509,  pages 89  (year 2011)NoStop
[Tóth et al.(2009)Tóth,
  Knapp, Gühne, and Briegel]ramsey2
  author author G. Tóth, author C. Knapp,
  author O. Gühne,  and  author H. J. Briegel, 10.1103/PhysRevA.79.042334 journal journal Phys. Rev. A volume 79, pages 042334 (year 2009)NoStop
[Sørensen et al.(2001)Sørensen, Duan, Cirac, and  Zoller]ramsey3
  author author A. Sørensen, author L.-M. Duan, author J. I. Cirac,   and author P. Zoller, 10.1038/35051038 journal journal
  Nature (London) volume 409, pages 63
  (year 2001)NoStop
[Duan et al.(2002)Duan,
  Cirac, and Zoller]ramsey5
  author author L.-M. Duan, author J. I. Cirac,   and author P. Zoller, 10.1103/PhysRevA.65.033619 journal journal Phys. Rev. A volume 65, pages 033619 (year 2002)NoStop
[Helmerson and You(2001)]you2001
  author author K. Helmerson and author L. You, 10.1103/PhysRevLett.87.170402 journal journal Phys. Rev. Lett. volume 87, pages 170402 (year
  2001)NoStop
[Zhang et al.(2017)Zhang,
  Zhou, Zhou, Guo, and  Zhou]zxf2017
  author author Y.-C. Zhang, author X.-F. Zhou,
  author X. Zhou, author
  G.-C. Guo,  and author
  Z.-W. Zhou, 10.1103/PhysRevLett.118.083604 journal journal
  Phys. Rev. Lett. volume 118, pages
  083604 (year 2017)NoStop
[Hu et al.(2017)Hu,
  Chen, Vendeiro, Urvoy,
  Braverman, and Vuleti ćć]jzhu2017
  author author J. Hu, author W. Chen, author Z. Vendeiro, author
  A. Urvoy, author B. Braverman,  and author V. Vuleti ćć, 10.1103/PhysRevA.96.050301 journal journal Phys. Rev. A volume 96, pages 050301(R) (year 2017)NoStop
[André and Lukin(2002)]tat1
  author author A. André and author M. D. Lukin, 10.1103/PhysRevA.65.053819 journal
  journal Phys. Rev. A volume 65,  pages 053819 (year 2002)NoStop
[Bouchoule and Mølmer(2002)]tat2
  author author I. Bouchoule and author K. Mølmer, 10.1103/PhysRevA.65.041803 journal journal Phys. Rev. A volume
  65, pages 041803(R) (year 2002)NoStop
[Zhang et al.(2003)Zhang,
  Helmerson, and You]tat4
  author author M. Zhang, author K. Helmerson,   and author L. You, 10.1103/PhysRevA.68.043622 journal journal Phys. Rev. A volume 68, pages 043622 (year 2003)NoStop
[André et al.(2002)André, Duan, and Lukin]tat6
  author author A. André, author L.-M. Duan,   and author M. D. Lukin, 10.1103/PhysRevLett.88.243602 journal journal Phys. Rev. Lett. volume 88, pages 243602 (year 2002)NoStop
[Wang et al.(2017)Wang,
  Qu, Li, Bao, Vuleti ćć, and Xiao]xiao2017
  author author M. Wang, author W. Qu, author P. Li, author
  H. Bao, author V. Vuleti ćć,  and  author Y. Xiao, 10.1103/PhysRevA.96.013823 journal journal Phys.
  Rev. A volume 96, pages 013823
  (year 2017)NoStop
[Takeuchi et al.(2005)Takeuchi, Ichihara, Takano, Kumakura, Yabuzaki, and Takahashi]oat2005
  author author M. Takeuchi, author S. Ichihara,
  author T. Takano, author M. Kumakura, author
  T. Yabuzaki,  and author
  Y. Takahashi, 10.1103/PhysRevLett.94.023003 journal journal
  Phys. Rev. Lett. volume 94, pages
  023003 (year 2005)NoStop
[Leroux et al.(2010)Leroux,
  Schleier-Smith, and Vuleti ćć]oat2010
  author author I. D. Leroux, author M. H. Schleier-Smith,  and author V. Vuleti ćć, 10.1103/PhysRevLett.104.073602 journal journal
  Phys. Rev. Lett. volume 104, pages
  073602 (year 2010)NoStop
[Gross et al.(2010)Gross,
  Zibold, Nicklas, Estève,  and Oberthaler]gross2010
  author author C. Gross, author T. Zibold,
  author E. Nicklas, author J. Estève,  and author M. K. Oberthaler, 10.1038/nature08919 journal journal
  Nature (London) volume 464, pages
  1165 (year 2010)NoStop
[Riedel et al.(2010)Riedel,
  Böhi, Li, Hänsch,
  Sinatra, and Treutlein]riedel2010
  author author M. F. Riedel, author P. Böhi,
  author Y. Li, author
  T. W. Hänsch, author
  A. Sinatra,  and author
  P. Treutlein, 10.1038/nature08988 journal journal Nature
  (London) volume 464, pages 1170
  (year 2010)NoStop
[Sørensen and Mølmer(2001)]ramsey4
  author author A. S. Sørensen and author K. Mølmer, 10.1103/PhysRevLett.86.4431 journal journal Phys. Rev. Lett. volume 86, pages 4431 (year
  2001)NoStop
[Law et al.(2001)Law,
  Ng, and Leung]law2001
  author author C. K. Law, author H. T. Ng,  and  author P. T. Leung, 10.1103/PhysRevA.63.055601 journal journal Phys. Rev. A volume 63, pages 055601 (year 2001)NoStop
[Jin and Kim(2007)]jin2007
  author author G.-R. Jin and author S. W. Kim,  10.1103/PhysRevLett.99.170405 journal
  journal Phys. Rev. Lett. volume 99,  pages 170405 (year 2007)NoStop
[Liu et al.(2011)Liu,
  Xu, Jin, and You]liu2011
  author author Y. C. Liu, author Z. F. Xu,
  author G. R. Jin,  and  author L. You, 10.1103/PhysRevLett.107.013601 journal journal
  Phys. Rev. Lett. volume 107, pages
  013601 (year 2011)NoStop
[Huang et al.(2015)Huang,
  Zhang, Zou, Zou, and  Guo]guo2015
  author author W. Huang, author Y.-L. Zhang,
  author C.-L. Zou, author X.-B. Zou,  and author G.-C. Guo, 10.1103/PhysRevA.91.043642 journal journal Phys.
  Rev. A volume 91, pages 043642
  (year 2015)NoStop
[Wu et al.(2015)Wu,
  Tey, and You]wln2015
  author author L.-N. Wu, author M. K. Tey,  and  author L. You, 10.1103/PhysRevA.92.063610 journal journal Phys.
  Rev. A volume 92, pages 063610
  (year 2015)NoStop
[Shen and Duan(2013)]duan2013
  author author C. Shen and author L.-M. Duan, 10.1103/PhysRevA.87.051801 journal
  journal Phys. Rev. A volume 87,  pages 051801(R) (year 2013)NoStop
[Bukov et al.(2018)Bukov,
  Day, Sels, Weinberg,
  Polkovnikov, and Mehta]phase1
  author author M. Bukov, author A. G. R. Day,
  author D. Sels, author
  P. Weinberg, author
  A. Polkovnikov,  and author
  P. Mehta, 10.1103/PhysRevX.8.031086 journal journal Phys.
  Rev. X volume 8, pages 031086
  (year 2018)NoStop
[Bukov(2018)]state1
  author author M. Bukov, 10.1103/PhysRevB.98.224305 journal
  journal Phys. Rev. B volume 98,  pages 224305 (year 2018)NoStop
[An and Zhou(2019)]dlzhou2019
  author author Z. An and author D. L. Zhou,  10.1209/0295-5075/126/60002 journal journal EPL (Europhysics Letters) volume
  126, pages 60002 (year 2019)NoStop
[sup()]supp2
  @noop  note See supplementary materials for a complete
  description of our DRL task, the DRL protocol with π/3 pulses, grid
  search optimization, IS and the DRL policy at N = 500, and a detailed
  discussion of robustness against pulse noise.Stop
[Dietterich(2000)]sparse1
  author author T. G. Dietterich, 10.1613/jair.639 journal
  journal Journal of Artificial Intelligence Research volume 13, pages 227 (year
  2000)NoStop
[Barto and Mahadevan(2003)]sparse2
  author author A. G. Barto and author S. Mahadevan, 10.1023/a:1022140919877 journal journal Discrete Event Dynamic Systems volume 13, pages 41 (year
  2003)NoStop
[Schulman et al.(2017)Schulman, Wolski, Dhariwal, Radford, and Klimov]ppo2017
  author author J. Schulman, author F. Wolski,
  author P. Dhariwal, author A. Radford,  and author O. Klimov, https://arxiv.org/pdf/1707.06347 journal journal
  arXiv:1707.06347  (year 2017), http://arxiv.org/abs/1707.06347 1707.06347 NoStop
[Opatrný(2015)]opatrny2015
  author author T. Opatrný, 10.1103/PhysRevA.91.053826 journal journal Phys. Rev. A volume
  91, pages 053826 (year 2015)NoStop
[Li et al.(2008)Li,
  Castin, and Sinatra]liyun2008
  author author Y. Li, author Y. Castin,  and  author A. Sinatra, 10.1103/PhysRevLett.100.210401 journal journal Phys. Rev. Lett. volume 100, pages 210401 (year 2008)NoStop
[Hosten et al.(2016)Hosten,
  Krishnakumar, Engelsen, and Kasevich]hosten2016
  author author O. Hosten, author R. Krishnakumar, author N. J. Engelsen,  and author M. A. Kasevich, 10.1126/science.aaf3397 journal
  journal Science volume 352, pages 1552 (year 2016)NoStop
[Haine(2018)]haine2018
  author author S. A. Haine, 10.1103/PhysRevA.98.030303 journal
  journal Phys. Rev. A volume 98,  pages 030303(R) (year 2018)NoStop





  Supplementary Material





This supplementary provides discussions that were left out of the main text due to space limitations. First, we briefly introduce the key concepts of deep reinforcement learning (DRL) and specify our optimization and search problem.
Then, we will give the results of a DRL task with π/3 pulses.
This is followed by introduction of grid search optimization for IS and comparative studies of squeezing coefficient between DRL policy and DRL-inspired scheme (IS).
In the end, we discuss the robustness of our policy against noise of the control pulses.




§ DEEP REINFORCEMENT LEARNING TASK

DRL task is modeled by a Markov decision process (MDP) as shown in Fig. <ref>. At each time step t, the agent obtains the observation s_t ∈𝒮 of the environment and select an action a_t ∈𝒜 based on current policy ϖ(a_t|s_t). The environment then evolves to s_t+1 due to action a_t and returns a scalar reward r_t∈ℛ back to the agent. Policy ϖ is consequently updated through such experiences data in some way to maximize accumulated rewards.



The definitions of state 𝒮, action 𝒜, and reward function ℛ in the spin squeezing problem are as follows.



  * 𝒮: For the collective spin system, we choose {𝐉,J_x^2,J_z^2} as the state representation instead of its wave function |ψ⟩, although the latter contains complete information. Using such relevant physical observables as state representation makes
    the policy directly generalizable and the final policy easily interpretable with clear physical insights. Here, the mean spin 𝐉=(J_x,J_y,J_z) provides the direction of the collective spin and {J_x^2, J_z^2}
    contains correlations between different atomic spins.


  * 𝒜: The action space consists of three discrete operations {0,±π/2}, where a=0 means evolving for a time interval δ t under bare OAT Hamiltonian H_OAT = χ J_z^2, while a=π/2(-π/2) means the same action 0 preceded by a π/2(-π/2) pulse.


  * ℛ: We use squeezing coefficient ξ^2=4(Δ𝐉_)_min^2/N as the object function to optimize for spin squeezing <cit.>. The total achievable reward is R_tot=ξ^2_i - ξ^2_f, where ξ^2_i = 1 for our chosen initial CSS and ξ_f^2 is the squeezing ratio at final time.
    To avoid the problem of sparse rewarding, we decompose the object function into a summation form:

    R_tot=∑_j=1^nξ^2(t_j-1) - ξ^2(t_j).

At each time step t_j, a reward r_j = ξ^2(t_j-1) - ξ^2(t_j), the instantaneous decrease of spin squeezing coefficient, is fed back to the agent. Dense rewards make the training process quicker and more stable. Furthermore, the reward can be modified into

    r_j →log_10 ( ξ^2(t_j-1)/ξ^2(t_j) ),

which prevents the decline of learning efficiency when squeezing approaches to the minimum.


The proximal policy optimization (PPO) algorithm is employed to find the optimized policy ϖ^* that can maximize the cumulative reward R,


    ϖ^* = max_ϖ R,   with   R=∑_jγ^tr_j,

where γ is a discount factor. It is typically chosen very close to 1 to avoid greedy solutions. The pseudo-code of PPO algorithm is shown in Table <ref> <cit.>. To facilitate the training process, we encapsulated the evolution of spin state into a gym environment as suggested by openAI <cit.>. In PPO algorithm, the policy is stochastic which returns a distribution on action space for a given state s. Here, in our problem on optimizing spin squeezing, the action space 𝒜 contains only three operations, so the policy reduces to a discrete distribution that satisfies

    ∑_a∈{ 0,±π/2 }ϖ(a|s) ≡ 1.




Once a policy is learnt, we can either choose the action with maximum probability as a=max_a'ϖ^*(a'|s) to give a deterministic protocol or obtain a protocol by selecting the best one among multiple protocols from a simple sampling based on ϖ^*(a'|s).

For a system with total particle number N=100, a simple neural network
is used to parameterize the actor and critic network in PPO.
Both of them contain two fully connected hidden layers with 32 and 16 neurons respectively.
Each learning episode is divided into 50 consecutive steps and the total evolution time is limited to χ t_c=0.08. The other hyper-parameters used in our training are listed in Table <ref>. No sophisticated hyper-parameter tuning is used in our problem.









§ DRL PROTOCOL WITH Π/3 PULSE FOR N = 100

In this section, we illustrate the underlying mechanism of the DRL policy in improving SS by using another form of pulse, with ∫^t_0 + Δ t_t_0Ω(t)dt = π/3. We call it 
π/3-policy in the following. In the π/3-pulse training task, all the hyper-parameters are chosen as same as the case of π/2-pulse (Table.<ref>).
The results for a system with N=100 are shown in Fig. <ref>.


By comparing the pulse sequences suggested by DRL policy in the π/2-policy (discussed in the main text) and the π/3-policy, we find that they share similar features: pulses come in pair and are applied around the instants when the effective TACT outperforms OAT or when the current squeezing is lagging behind that of OAT.




For the π/3 case, the time evolution operator U for a (±π/3,∓π/3) pulse pair reads,

    U = e^± iπ/3J_ye^-iχ J_z^2δ te^∓ iπ/3J_y = e^-iχ(J_xsin(π/3) ± J_zcos(π/3))^2δ t≃ e^-i3/4χ J_x^2δ te^∓ i√(3)/4χ(J_xJ_z + J_zJ_x)δ te^-i1/4χ J_z^2δ t,

where we have neglected the high-order 𝒪(δ t^2) term in the last equality.
Again, we find the non-linear term ∝ J_x^2, which contributes to the rectification of the over-twisting problem as discussed in the main text.  Note that here we have additional crossing term J_xJ_z+J_zJ_x, which is absent in the π/2-pulse case and degrades the rectification effect. It can be partly eliminated by alternatively and consecutively applying (π/3,-π/3) and (-π/3,π/3), whose crossing terms have opposite signs. 












§ GRID SEARCH FOR IS AND COMPARISON BETWEEN DRL AND IS

In our IS, the whole evolution is also divided into M steps with equal duration δ t_0. Initially, the time duration δ t between ±π/2 pulses in a pulse pair is set as δ t_0 and the pulse pair applying moment is determined by our concluded rules from DRL protocols. However, the direct implement of IS in a N=500 system (open squares with black dotted guiding line in Fig. <ref>(a1)) cannot reach our expectation, performing as well as DRL policy (green filled circles with solid guiding line in Fig. <ref>(b)), since the DRL policy can fine tune the pulse applying moments around the instants we concluded. We noticed that the time duration δ t in a pulse pair directly determines the non-linear rectification degree on OAT induced over-twisting state, such that the performance of our IS can be enhanced via optimizing δ t. This optimization process is done through a so-called grid search (GS) calculation. That is, we scan the δ t ∈ [0,δ t_0] and figure out an optimal δ t leading to the best performance. The searching process is shown in Fig. <ref>(a2) and the optimized IS result is compared with OAT and effective TACT in Fig. <ref>(a1). After a grid search optimization, the squeezing performance of our IS is approaching to TACT.

To demonstrate that we have captured the key characteristic features of the DRL policy, we first compare the squeezing coefficients obtained with DRL and IS for N = 500 as shown in Fig. <ref>(b). For the DRL policy, the maximum number of pulse pairs is limited to 4 and the action space 𝒜 is contracted to {0,1}, where `1' means applying a pulse pair (π/2,-π/2) and `0' means free evolution under bare OAT Hamiltonian for δ t.

The optimal squeezing for both policies reach the same extreme level of the effective TACT, with DRL approaching the optimal squeezing at an earlier moment. Taking the computation effort into account, IS is obviously favored. The lower panel of Fig. <ref> shows the corresponding pulse sequence.
Out of four pulse pairs, three are found to be applied essentially at the same moments for these two policies.






§ ROBUSTNESS AGAINST FLUCTUATIONS IN CONTROL PULSES

In previous discussion, our numerical calculations for IS always assume that the ±π/2 control pulses are executed
so quickly that during it the OAT interaction is negligible.

Typically, for a 10 kHz microwave Rabi frequency, the duration of a π/2 pulse is about 25 μs, which translates into about 10% of the time duration δ t between pulse for N = 2000 and χ = 2π× 0.063 Hz in ^87Rb <cit.>. In Fig. <ref>(a), phase accumulation from the bare OAT Hamiltonian over the applied pulse is considered. It will lead to a degradation of the achievable spin squeezing (the orange dash-dotted line in Fig. <ref>(a)), although some of the squeezing performance can be rectified after a δ t optimization with respect to real pulse operation (the green dash-dotted line in Fig. <ref>(a)).

Next, we consider imperfections of the pulse area for a square-shaped versus a Gaussian-shaped pulse we used. Due to the associated fluctuation phase mismatch of the ±π/2 pulse in a pair, the mean spin will be rotated along the y-axis, i.e. 𝐉 = (J_x,0,0) → (J_xcos(θ_r),0,J_xsin(θ_r)), where θ_r is the net non-ideal cumulative phase during a pulse pair.
The subsequent squeezing of such a state is dramatically degraded under bare OAT Hamiltonian evolution. We have numerically simulated
such imperfections using π/2 pulses with 0.05% and 0.01% Gaussian stochastic uncertainties and the squeezing coefficient
is estimated assuming that
 mean spin still points along the x-axis 𝐉 = (J_x,0,0).
The squeezing coefficient ξ^2 shown in Fig. <ref>(b) is sampled at χ t = 0.0117,
corresponding to the optimal squeezing position of our IS at N =2000 system (the red open circle mark in Fig. <ref>(a)).
The squeezing coefficient ξ^2 under such a fluctuating error distribution to the ±π/2
pulse area in most of the samples is found to still outperform that of OAT.



4
natexlab#1#1bibnamefont#1#1bibfnamefont#1#1citenamefont#1#1url<#>1urlprefixURL 


[Kitagawa and Ueda(1993)]ueda1993
authorM. Kitagawa and
  authorM. Ueda,
  journalPhys. Rev. A volume47,
  pages5138 (year1993),
  <https://link.aps.org/doi/10.1103/PhysRevA.47.5138>.

[ppo()]ppo
howpublishedOpenAI Spinning Up,
  <https://spinningup.openai.com/>.

[gym()]gym
howpublishedOpenAI Gym, <https://gym.openai.com/>.

[Gross et al.(2010)Gross, Zibold,
  Nicklas, Estève, and Oberthaler]gross2010
authorC. Gross,
  authorT. Zibold,
  authorE. Nicklas,
  authorJ. Estève,
  and authorM. K.
  Oberthaler, journalNature (London)
  volume464, pages1165 (year2010),
  ISSN issn1476-4687,
  <http://dx.doi.org/10.1038/nature08919>.


