Generalized Quantum Reinforcement Learning with
Quantum Technologies
F. A. Cárdenas-López1,2,† , L. Lamata3 , J. C. Retamal1,2 , and E. Solano3,4

arXiv:1709.07848v1 [quant-ph] 22 Sep 2017

1 Departamento

de Fı́sica, Universidad de Santiago de Chile (USACH), Avenida Ecuador 3493, 9170124, Santiago,
Chile
2 Center for the Development of Nanoscience and Nanotechnology 9170124, Estación Central, Santiago, Chile
3 Department of Physical Chemistry, University of the Basque Country UPV/EHU, Apartado 644, 48080 Bilbao,
Spain
4 IKERBASQUE, Basque Foundation for Science, Maria Dı́az de Haro 3, 48013 Bilbao, Spain
† francisco.cardenas@usach.cl

ABSTRACT
We propose a protocol to perform generalized quantum reinforcement learning with quantum technologies. At variance with
recent results on quantum reinforcement learning with superconducting circuits [L. Lamata, Sci. Rep. 7, 1609 (2017)], in our
current protocol coherent feedback during the learning process is not required, enabling its implementation in a wide variety of
quantum systems. We consider diverse possible scenarios for an agent, an environment, and a register that connects them,
involving multiqubit and multilevel systems, as well as open-system dynamics. We finally propose possible implementations
of this protocol in trapped ions and superconducting circuits. The field of quantum reinforcement learning with quantum
technologies will enable enhanced quantum control, as well as more efficient machine learning calculations.

Keywords: Quantum Machine learning, Quantum Reinforcement Learning, Trapped Ions, Superconducting Circuits.

Introduction
Machine Learning (ML) is a subfield of Artificial Intelligence (AI) that has attracted increasing attention in the last years. ML
usually refers to a computer program which can learn from experience E with respect to some class of task T and performance
measure P, if its performance at tasks in T, as measured by P, improves with experience E1 . In other words, Machine Learning
addresses the problem of how a computer algorithm can be constructed to automatically improve with experience. Several
applications in this field have been implemented such as handwriting pattern recognition2 , speech recognition3 and the
development of a computer able to beat an expert Go player4 , just to name a few.
The learning process in ML can be divided in three types: supervised learning, unsupervised learning and reinforcement
learning5 . In supervised machine learning, an initial data set has the function of training the system for later prediction
making or to classify data. Usually, supervised learning problems are categorized into regression (continuous output) or
classification (discrete output). Unsupervised learning allows one to address problems where the training data is not necessary
and only correlations between subsets in the data (clustering) are considered and analyzed. Finally, reinforcement learning6
differs from supervised and unsupervised learning in that it takes into account a scalar parameter (reward) to evaluate the
input-output relation in a trial and error way. In this case, the system (so-called “agent”) obtains information from its outer
world (“environment”) to decide which is the better way to optimize itself, for adapting to the environment.
Quantum information processing (QIP) has contributed positively in the development of the machine learning field, with
several quantum algorithms for machine learning with significant possible gains with respect to their classical counterparts7–11 .
More specifically, quantum algorithms have been developed and in some cases implemented for supervised and unsupervised
learning problems12–16 . However, quantum reinforcement learning has not been widely explored and just a few results have
been obtained up to now17–21 . Related topics in biomimetic quantum technologies are quantum memristors22–25 , as well as
quantum Helmholtz and Boltzmann machines26–28 .
The field of quantum technologies has grown extensively in the past decade, being two of its most developed architectures
trapped ions29, 30 and superconducting circuits31–33 . Current technological progress in trapped ions has allowed us to implement
quantum protocols with several ions involving high-fidelity single and two-qubit gates as well as high-fidelity readout34, 35 .
Superconducting circuits have also proven to be an excellent platform to perform quantum information processing protocols
because of their individual addressing and scalability. Two-qubit quantum gates have achieved fidelities larger than 99%36, 37 in

1

this platform. Furthermore, technological progress in this architecture has made possible to build artificial atoms with high
coherence time in coplanar38 and 3D architecture39 , allowing for the development of feedback control with superconducting
circuits40, 41 . This feedback mechanism has inspired protocols for quantum reinforcement learning with superconducting
circuits21 where the feedback loop control allows one to reward and restart the system to obtain maximal learning fidelity.
Here, we propose a protocol to perform generalized quantum reinforcement learning with quantum technologies. In
this sense, and at variance with a previous result21 , we extend the realm of the quantum reinforcement learning protocol to
multiqubit, multilevel, and open quantum systems, therefore permitting a wider set of scenarios. Our protocol considers a
quantum system (the agent), which interacts with an external system (its environment) via an auxiliary system (a register). The
aim of our quantum reinforcement learning protocol is for the agent to acquire information from its environment and adapt
to it, via a rewarding mechanism. In our specific case, the quantum agent aims at attaining maximum quantum state overlap
with the environment state, in the sense that local measurements on agent and environment will produce the same outcomes or,
equivalently, that the agent and environment entangled final state is invariant under the exchange of these two subsystems. An
interpretation of this outcome is that the agent has learned the environment state and consequently modified it. We analyze the
case where the register subspace is larger than agent and environment subspaces. The inclusion of more elements in the register
subspace allows for delaying the application of the rewarding criterion to the end of the quantum protocol. This fact will
enable its implementation in a wider variety of quantum platforms, besides superconducting circuits with coherent feedback.
We also study quantum reinforcement learning in the case where agent, environment and register are composed of qudits. In
this case, we obtain that the maximal learning fidelity is achieved in a fixed number of steps in the qudit dimension, and this
number scales polynomially with the number of subsystems in the environment subspace. In addition, we analyse quantum
reinforcement learning in the situation where the environment is larger than the agent. We highlight two results: the first of them
is obtained when considering that the register has the same elements than the environment. In this case, two rewarding criteria
are needed to obtain maximal learning fidelity and the entanglement between the agent and a specific part of the environment is
a key resource. The other case is the situation where the register has more elements than the environment. In this case, only
one measurement is needed to obtain maximal learning fidelity and the environment-agent entanglement is not a key resource.
Based on this fact, the rewarding criterion is applied at the end of the protocol. Finally, we describe how our quantum learning
protocols can be implemented in quantum platforms as trapped ions and superconducting circuits.

Quantum Reinforcement Learning Protocol with Only One Final Measurement
Here, we introduce a protocol to perform quantum reinforcement learning, which introduces significant novelties with respect
to the existing literature. Unlike a previous quantum reinforcement learning result21 , the protocol described here needs only
one measurement at the end of the procedure, allowing for its implementation in a variety of quantum platforms including ions
and photons. The improvement relies on adding more registers than before21 and making them interact conditionally with each
other. The inclusion of ancillary systems has proven to be useful in several implementations of quantum information, because
measurements on the ancillary system allow one in principle to obtain information about the main system without destroying it.
Moreover, the measurement associated with the rewarding criterion is performed at the end of the protocol. This opens the
possibility to implement quantum reinforcement learning protocols in architectures for which implementing coherent feedback
may be a challenging problem.
The quantum reinforcement learning protocol described here works in the following way. We firstly consider an agent and
environment, composed of one qubit each, and two register qubits, see Fig. 1. The first step is to encode the environment
information in the register states (usually this kind of operation in the context of classical reinforcement learning is called
the action). Subsequently, the internal states of the registers are conditionally updated given the information obtained from
the agent (usually this kind of operation in classical reinforcement learning is called the percept). Finally, an agent-register
interaction updates the agent state (partial rewarding mechanism). At this stage the rewarding criterion is satisfied, in the form
of a maximally correlated agent-environment state, in the sense that local measurements on agent and environment will produce
the same outcomes. On the other hand, the agent-environment system is also entangled with the two registers, and in order to
attain a correlated pure state of agent and environment, a single, final measurement may be performed on the two register states.
This will produce an agent-environment state maximizing the learning fidelity defined as FAE = |hψA |φE i|, where |ψA i is the
agent state and |φE i is the environment state.
To perform our quantum reinforcement learning protocol we consider that initially agent and environment are in arbitrary
single-qubit pure states, whereas the register states are in their ground state, namely
{|Ai = αA0 |0iA + αA1 |1iA , |Ei = αE0 |0iE + αE1 |1iE , |Ri = |0i1 |0i2 }

(1)

|Ψi0

(2)

= |Ai|Ei|Ri.

The first step in the protocol is to extract information from the environment, updating the information in the registers
conditionally to the environment state. This process is done by applying a pair of CNOT gates in the environment-register
2/16

A
E
R1
R2
Figure 1. Proposed protocol to perform quantum reinforcement learning with final measurement. We consider a set composed
of four qubits, corresponding to agent A, environment E, and registers R1 and R2 . The considered interactions agent-register,
register-register and environment-register consist of CNOT gates. The measurement in the register subspace is denoted by the
rightmost box.
subspace. Here, the first system is the control and the second the target,
|Ψi1
|Ψi1

CNOT CNOT
= U(E,R
U
|Ψi0 ,
2 ) (E,R1 )


0
1
= αA |0iA + αA |1iA αE0 |0iE |01 , 02 i + αE1 |1iE |11 , 12 i .

(3)
(4)

Then, the information encoded on the registers is updated conditional on the agent state. As the register subspace is larger than
the agent subspace, we will choose which part of the register subspace will the agent update. Without loss of generality, let us
assume that the register R1 will be updated. The upgrade of agent subspace is performed by a CNOT gate acting in the A − R1
subspace, where the agent state is the control and the register is the target,
|Ψi2

CNOT
= U(A,R
|Ψi1 ,
1)

|Ψi2

=


αA0 αE0 |0iA |0iE |0i1 |0i2 + αA0 αE1 |0iA |1iE |1i1 |1i2 + αA1 αE0 |1iA |0iE |1i1 |0i2 + αA1 αE1 |1iA |1iE |0i1 |1i2 .

(5)

Subsequently, the register R2 is also updated with respect to the R1 state. This is accomplished by applying a CNOT gate in the
register subspace, where R1 acts as control and R2 as target,
|Ψi3

CNOT
= U(R
|Ψi2 ,
1 ,R2 )

|Ψi3

=


αA0 αE0 |0iA |0iE |0i1 |0i2 + αA0 αE1 |0iA |1iE |1i1 |0i2 + αA1 αE0 |1iA |0iE |1i1 |1i2 + αA1 αE1 |1iA |1iE |0i1 |1i2 .

(6)

Followingly, we update the agent state according to the information encoded in the register R1 . This is done by applying a
CNOT gate in the R1 − A subspace, where R1 is the control and A is the target,
|Ψi4

CNOT
= U(R
|Ψi3 ,
1 ,A)

|Ψi4

=


αA0 αE0 |0iA |0iE |0i1 |0i2 + αA0 αE1 |1iA |1iE |1i1 |0i2 + αA1 αE0 |0iA |0iE |1i1 |1i2 + αA1 αE1 |1iA |1iE |0i1 |1i2 .

(7)

We point out that, in the previous state, agent and environment are already maximally correlated, in the sense of having the
same outcomes with respect to local measurements performed on either of them, or, equivalently, the state is invariant under
particle exchange with respect to the agent-environment subsystem. We also remark that this state is general, valid for any
initial agent and environment states. The fact that agent and environment get entangled with the two registers allows one to
distinguish between identical agent-environment components that originate from different initial states, namely, to distinguish
between states arising from αA0 αE0 or αA1 αE0 , as well as from αA0 αE1 or αA1 αE1 .
Finally, by performing a projective measurement on the register subspace, the rewarding criteron is satisfied. It is easy to
show that, independently of the measurement outcome, the learning fidelity FAE = |hψA |φE i| is maximal. In this case only
one iteration of the protocol is sufficient in order that the agent adapts to the environment. Moreover, throughout the protocol,
measurements on agent and/or environment are not required, which may allow its implementation in a variety of quantum
platforms as trapped ions, superconducting circuits, and quantum photonics.
3/16

A
E
R1
R2
R3
R4
Figure 2. Schematic representation of quantum reinforcement learning protocol for multiqubit systems. Agent, environment
and registers are denoted as A, E and R1 , R2 , R3 and R4 , respectively. The measurement in the register subspace is denoted by
the rightmost box.

Quantum Reinforcement Learning for Multiqubit Systems with Final Measurement
In the previous section, we have showed that by considering more than just one register the rewarding criterion in the quantum
reinforcement learning algorithm can be done at the end of our protocol. The same results can be obtained when we consider
more complex configurations. Indeed, by assuming that agent and register are composed of two qubits each, and four qubits act
as registers, we show that the rewarding criterion can also be applied at the end of the quantum protocol. Let us illustrate this
fact with an analysis for multiqubit agent, environment, and register states,
|Ai = αA00 |00iA + αA01 |01iA + αA10 |10iA + αA11 |11iA ,

(8)

|Ei = αE00 |00iE

(9)

+ αE01 |01iE

+ αE10 |10iE

+ αE11 |11iE ,

|Ri = |0i1 |0i2 |0i3 |0i4 ,

(10)

|Ψi0 = |Ai|Ei|Ri.

(11)

Following the same procedure described previously, the protocol consists mainly in three types of interaction, as shown in
Fig. 2. Firstly, we update the registers conditionally to the environment states. More specifically, we consider an interaction
between the environment qubits E1 and E2 with the registers R1 and R2 , respectively. In this description, the environment acts
as control and the registers act as targets in the CNOT gates,
|Ψi1

CNOT CNOT
= U(E
U
, |Ψi0 ,
1 ,R1 ) (E2 ,R2 )

|Ψi1


= |Ai αE00 |00iE |0i1 |0i2 |0i3 |0i4 + αE01 |01iE |0i1 |1i2 |0i3 |0i4 + αE10 |10iE |1i1 |0i2 |0i3 |0i4 + αE11 |11iE |1i1 |1i2 |0i3 |0i4 .
(12)

Thereafter, we update similarly the remaining registers, that is, we apply a CNOT gate between the environment qubits E1 and
E2 and the register qubits R3 and R4 , respectively, obtaining
|Ψi2

CNOT CNOT
= U(E
U
|Ψi1 ,
1 ,R3 ) (E2 ,R4 )

|Ψi2


= |Ai αE00 |00iE |0i1 |0i2 |0i3 |0i4 + αE01 |01iE |0i1 |1i2 |0i3 |1i4 + αE10 |10iE |1i1 |0i2 |1i3 |0i4 + αE11 |11iE |1i1 |1i2 |1i3 |1i4 .
(13)

4/16

Next step consists in updating a part of the register subspace conditionally to the agent state. Thus, the registers R1 and R2 will
be updated via A1 and A2 , respectively,
|Ψi3

CNOT CNOT
= U(A
U
|Ψi2 ,
1 ,R1 ) (A2 ,R2 )

|Ψi3

= αA00 αE00 |00iA |00iE |0i1 |0i2 |0i3 |0i4 + αA00 αE01 |00iA |01iE |0i1 |1i2 |0i3 |1i4 + αA00 αE10 |00iA |10iE |1i1 |0i2 |1i3 |0i4
+αA00 αE11 |00iA |11iE |1i1 |1i2 |1i3 |1i4 + αA01 αE00 |01iA |00iE |0i1 |1i2 |0i3 |0i4 + αA01 αE01 |01iA |01iE |0i1 |0i2 |0i3 |1i4
+αA01 αE10 |01iA |10iE |1i1 |1i2 |1i3 |0i4 + αA01 αE11 |01iA |11iE |1i1 |0i2 |1i3 |1i4 + αA10 αE00 |10iA |00iE |1i1 |0i2 |0i3 |0i4
+αA10 αE01 |10iA |01iE |1i1 |1i2 |0i3 |1i4 + αA10 αE10 |10iA |10iE |0i1 |0i2 |1i3 |0i4 + αA10 αE11 |10iA |11iE |0i1 |1i2 |1i3 |1i4
+αA11 αE00 |11iA |00iE |1i1 |1i2 |0i3 |0i4 + αA11 αE01 |11iA |01iE |1i1 |0i2 |0i3 |1i4 + αA11 αE10 |11iA |10iE |0i1 |1i2 |1i3 |0i4
+αA00 αE11 |11iA |11iE |0i1 |0i2 |1i3 |1i4 .

(14)

Afterwards, to obtain orthogonal outcomes in the register subspace we perform a pair of CNOT gates in this subspace. The
interaction will be between the registers that interact with a common environment, namely, register R1 interacts with R3 because
both have interacted with E1 . Similarly for R2 and R4 , which have interacted with E2 . In this case, R1 (R2 ) is the control and
R3 (R4 ) is the target.
|Ψi4

CNOT CNOT
= U(R
U
|Ψi3 ,
1 ,R3 ) (R2 ,R4 )

|Ψi4

= αA00 αE00 |00iA |00iE |0i1 |0i2 |0i3 |0i4 + αA00 αE01 |00iA |01iE |0i1 |1i2 |0i3 |0i4 + αA00 αE10 |00iA |10iE |1i1 |0i2 |0i3 |0i4
+αA00 αE11 |00iA |11iE |1i1 |1i2 |0i3 |0i4 + αA01 αE00 |01iA |00iE |0i1 |1i2 |0i3 |1i4 + αA01 αE01 |01iA |01iE |0i1 |0i2 |0i3 |1i4
+αA01 αE10 |01iA |10iE |1i1 |1i2 |0i3 |1i4 + αA01 αE11 |01iA |11iE |1i1 |0i2 |0i3 |1i4 + αA10 αE00 |10iA |00iE |1i1 |0i2 |1i3 |0i4
+αA10 αE01 |10iA |01iE |1i1 |1i2 |1i3 |0i4 + αA10 αE10 |10iA |10iE |0i1 |0i2 |1i3 |0i4 + αA10 αE11 |10iA |11iE |0i1 |1i2 |1i3 |0i4
+αA11 αE00 |11iA |00iE |1i1 |1i2 |1i3 |1i4 + αA11 αE01 |11iA |01iE |1i1 |0i2 |1i3 |1i4 + αA11 αE10 |11iA |10iE |0i1 |1i2 |1i3 |1i4
+αA00 αE11 |11iA |11iE |0i1 |0i2 |1i3 |1i4 .

(15)

Finally, we update the agent considering the states of the register in order that the rewarding criterion is satisfied. This is done
by applying two CNOT gates in the agent-register subspace, where A1 is controlled by R1 and A2 is controlled by R2 ,
|Ψi5

CNOT CNOT
= U(R
U
|Ψi4 ,
1 ,A1 ) (R2 ,A2 )

|Ψi5

= αA00 αE00 |00iA |00iE |0i1 |0i2 |0i3 |0i4 + αA00 αE01 |01iA |01iE |0i1 |1i2 |0i3 |0i4 + αA00 αE10 |10iA |10iE |1i1 |0i2 |0i3 |0i4

(16)

+αA00 αE11 |11iA |11iE |1i1 |1i2 |0i3 |0i4 + αA01 αE00 |00iA |00iE |0i1 |1i2 |0i3 |1i4 + αA01 αE01 |01iA |01iE |0i1 |0i2 |0i3 |1i4
+αA01 αE10 |10iA |10iE |1i1 |1i2 |0i3 |1i4 + αA01 αE11 |11iA |11iE |1i1 |0i2 |0i3 |1i4 + αA10 αE00 |00iA |00iE |1i1 |0i2 |1i3 |0i4
+αA10 αE01 |01iA |01iE |1i1 |1i2 |1i3 |0i4 + αA10 αE10 |10iA |10iE |0i1 |0i2 |1i3 |0i4 + αA10 αE11 |11iA |11iE |0i1 |1i2 |1i3 |0i4
+αA11 αE00 |00iA |00iE |1i1 |1i2 |1i3 |1i4 + αA11 αE01 |01iA |01iE |1i1 |0i2 |1i3 |1i4 + αA11 αE10 |10iA |10iE |0i1 |1i2 |1i3 |1i4
+αA00 αE11 |11iA |11iE |0i1 |0i2 |1i3 |1i4 .

(17)

From the latter Eq. (17), it is straightforward to see that independently of the measurement outcomes the learning fidelity is
maximal. Moreover, as in the previous case, one iteration of the quantum reinforcement protocol is needed to obtain maximal
learning fidelity, FAE = |hψA |φE i|.

Quantum Reinforcement Learning for Qudit Systems
So far, we have studied quantum reinforcement learning processes only for two-level systems or in pairs of them. However,
there are several quantum systems which cannot be described in terms of a two-level system. For instance, quantum harmonic
oscillators, electronic energy levels in an ion, and superconducting artificial atoms such as transmons42 , where for some regimes
of Josephson energy they must be considered as a three-level system. In this context, it is interesting to extend the quantum
reinforcement learning protocol developed here for cases where multilevel systems compound the agent, environment, and
register.
To perform the previous task, we first need to define a set of logic operations that we will perform on our system. In the
qubit case, the main logical operation applied is the CNOT gate, which considers a conditional interaction between two qubits,
where one acts as a control while the other acts as a target. The control qubit remains unchanged whereas the target qubit output
is modified by the addition modulo 2. Then, it is wise to assume that the set of logic operations between multilevel systems
could be defined in terms of an addition modulo D, where D stands for the dimension of one subsystem (agent, environment or
register subspaces), according to
U|ii1 | ji2 = |ii1 |i ⊕ ji2 .

(18)
5/16

A
E
R1
R2
Figure 3. Quantum reinforcement learning protocol for qudits. The systems involved are denoted as agent A, environment E
and registers R1 , R2 . In this case, the logical quantum gates which are applied in the learning protocol correspond to GXOR
gates. The measurement process in the register subspace is denoted with the rightmost box.
Here, i ⊕ j stands for the addition modulo D. This gate is usually known as XOR gate43 . For two-dimensional systems, this
gate corresponds to the CNOT gate. Nevertheless, for higher dimensional systems this definition presents several disadvantages.
For instance, the XOR gate defined as in Eq. (18) is unitary but not Hermitian for D > 2. Moreover, this logical operation is no
longer its own inverse. To avoid these problems, in the literature43 the generalized XOR gate (GXOR) has been defined as
GXOR1,2 |ii1 | ji2 = |ii1 |i

ji2 ,

(19)

where the operation denotes the difference i − j modulo D. The GXOR gate of Eq. (19) does not present the disadvantages
pointed out in the definition of Eq. (18). That is, the GXOR gate is Hermitian, unitary and i j = 0 only when i = j.
Considering our proposed protocol for single-qubit cases, we show that when we take into account multilevel systems, the
number of interactions to obtain maximal learning fidelity is fixed and depends only on the number of agent subsystems in the
protocol. Let us illustrate this with an example of multilevel agent-environment-register state,
N−1 N−1

∑ ∑ αAn αEm |niA |miE |0i1 |0i2 .

|Ψ0 i =

(20)

n=0 m=0

The first step in our protocol is identical to the equivalent one in the single-qubit case. We update the register conditionally on
the environment state, that is, we transfer information of the environment and encode it in the register system. This is done by
applying a pair of GXOR gates acting in the environment-register subsystem. In this case, the environment interacts with both
registers R1 and R2 . The environment acts as control and both registers are targets,
GXOR
|Ψ1 i = U(E,R
|Ψ0 i,
1)
N−1 N−1

∑ ∑ αAn αEm |niA |miE |mi1 |0i2 .

|Ψ1 i =

(21)

n=0 m=0

GXOR
|Ψ2 i = U(E,R
|Ψ1 i,
2)
N−1 N−1

∑ ∑ αAn αEm |niA |miE |mi1 |mi2 .

|Ψ2 i =

(22)

n=0 m=0

Once the information has been transferred to the register, we update the register R1 based on the agent state. That is, we
perform a GXOR gate in the subspace composed of agent and register. Here, the agent act as a control and the register R1 is the
target,
|Ψ3 i

GXOR
= U(A,R
|Ψ2 i,
1)

|Ψ3 i

=

N−1 N−1

∑ ∑ αAn αEm |niA |miE |n

mi1 |mi2 .

(23)

n=0 m=0

6/16

Orthogonal outcome measurements in the register subspace are provided by interactions between the registers in this subspace.
Thus, we apply a GXOR gate in the register subspace, where R1 is the control and R2 is the target,
|Ψ4 i

GXOR
= U(R
|Ψ3 i,
1 ,R2 )

|Ψ4 i

=

N−1 N−1

∑ ∑ αAn αEm |niA |miE |n

mi1 |(n

mi2 .

m)

(24)

n=0 m=0

Subsequently, the agent state is updated conditionally to the information encoded in the state of the register R1 . The GXOR
gate is applied in the register-agent subspace. In this case, R1 is the control and the agent is the target,
|Ψ5 i

GXOR
= U(R
|Ψ4 i,
1 ,A)

|Ψ5 i

=

N−1 N−1

∑ ∑ αAn αEm |0

miA |miE |n

mi1 |n

2mi2 .

(25)

n=0 m=0

For the case where the multi-level system contains D = 2, we recover the result discussed previously because of 0 m = m
for that dimension. On the other hand, we are interested in systems with more energy levels, such that we need to adapt the
protocol to obtain maximal learning fidelity for a fixed number of steps. In this case, we will update the agent subsystem by an
iterative interaction with registers R1 and R2 as shown in Fig. 3. Here, the agent always acts as target, while the registers are the
controls. Therefore, we apply a GXOR gate between the register R2 and the agent,
GXOR
|Ψ6 i = U(R
|Ψ5 i,
2 ,A)
N−1 N−1

|Ψ6 i =

∑ ∑ αAn αEm |n

miA |miE |n

mi1 |n

2mi2 .

(26)

n=0 m=0

Now, by applying a GXOR gate between the register R1 and the agent we obtain,
GXOR
|Ψ7 i = U(R
|Ψ6 i,
1 ,A)
N−1 N−1

|Ψ7 i =

∑ ∑ αAn αEm |0iA |miE |n

mi1 |n

2mi2 .

(27)

n=0 m=0

We perform subsequently a GXOR gate in the subspace composed of R2 and agent A,
GXOR
|Ψ8 i = U(R
|Ψ7 i,
2 ,A)
N−1 N−1

|Ψ8 i =

∑ ∑ αAn αEm |n

2miA |miE |n

mi1 |n

2mi2 .

(28)

n=0 m=0

Finally, applying a GXOR gate on the register-agent subspace we obtain the desired result. By considering a fixed number of
interactions between the set of agent, environment and register, the learning fidelity becomes maximal independently of the
outcome measurement on the register subspace, which can again be carried out at the end of the protocol,
GXOR
|Ψ9 i = U(R
|Ψ8 i,
1 ,A)
N−1 N−1

|Ψ9 i =

∑ ∑ αAn αEm |miA |miE |n

mi1 |n

2mi2 .

(29)

n=0 m=0

Thus, in a machine learning protocol where the learning units are composed by multilevel systems (see Fig 3), the number of
logical operations required to obtain maximal learning fidelity does not depend on the system dimension.
Example
Here, we exemplify how our reinforcement learning protocol works in qudit systems. We consider, without loss of generality,
the case for dimension D = 4. In this case, the agent-environment-register state has the following form,
{|Ai = αA0 |0i + αA1 |1i + αA2 |2i + αA3 |3i, |Ei = αE0 |0i + αE1 |1i + αE2 |2i + αE3 |3i, |Ri = |0i1 |0i2 }

(30)

|Ψi0 = |Ai|Ei|Ri.

(31)

7/16

As mentioned previously, the considered quantum gate is a GXOR gate with subtraction modulo 4. The first step is to update
the register according to the environment information,
|Ψi1

GXOR
= U(E,R
|Ψi0 ,
1)

|Ψi1

= (αA0 |0i + αA1 |1i + αA2 |2i + αA3 |3i)(αE0 |0iE |0i1 |0i2 + αE1 |1iE |1i1 |0i2 + αE2 |2iE |2i1 |0i2 + αE3 |3iE |3i1 |0i2 ),
(32)

|Ψi2

GXOR
= U(E,R
|Ψi1 ,
2)

|Ψi2

= (αA0 |0i + αA1 |1i + αA2 |2i + αA3 |3i)(αE0 |0iE |0i1 |0i2 + αE1 |1iE |1i1 |1i2 + αE2 |2iE |2i1 |2i2 + αE3 |3iE |3i1 |3i2 ).
(33)

Subsequently, the register is updated conditional to the agent state,
|Ψi3

GXOR
= U(A,R
|Ψi2 ,
1)

|Ψi3

= αA0 αE0 |0iA |0iE |0i1 |0i2 + αA0 αE1 |0iA |1iE |3i1 |1i2 + αA0 αE2 |0iA |2iE |2i1 |2i2 + αA0 αE3 |0iA |3iE |1i1 |3i2
+αA1 αE0 |1iA |0iE |1i1 |0i2 + αA1 αE1 |1iA |1iE |0i1 |1i2 + αA1 αE2 |1iA |2iE |3i1 |2i2 + αA1 αE3 |1iA |3iE |2i1 |3i2
+αA2 αE0 |2iA |0iE |2i1 |0i2 + αA2 αE1 |2iA |1iE |1i1 |1i2 + αA2 αE2 |2iA |2iE |0i1 |2i2 + αA2 αE3 |2iA |3iE |3i1 |3i2
+αA3 αE0 |3iA |0iE |3i1 |0i2 + αA3 αE1 |3iA |1iE |2i1 |1i2 + αA3 αE2 |3iA |2iE |1i1 |2i2 + αA3 αE3 |3iA |3iE |0i1 |3i2 .

(34)

Then, to obtain orthogonal outcome measurements in the register basis, we perform an interaction in the register subspace,
|Ψi4

GXOR
= U(R
|Ψi3 ,
1 ,R2 )

|Ψi4

= αA0 αE0 |0iA |0iE |0i1 |0i2 + αA0 αE1 |0iA |1iE |3i1 |2i2 + αA0 αE2 |0iA |2iE |2i1 |0i2 + αA0 αE3 |0iA |3iE |1i1 |2i2
+αA1 αE0 |1iA |0iE |1i1 |1i2 + αA1 αE1 |1iA |1iE |0i1 |3i2 + αA1 αE2 |1iA |2iE |3i1 |1i2 + αA1 αE3 |1iA |3iE |2i1 |3i2
+αA2 αE0 |2iA |0iE |2i1 |2i2 + αA2 αE1 |2iA |1iE |1i1 |0i2 + αA2 αE2 |2iA |2iE |0i1 |2i2 + αA2 αE3 |2iA |3iE |3i1 |0i2
+αA3 αE0 |3iA |0iE |3i1 |3i2 + αA3 αE1 |3iA |1iE |2i1 |1i2 + αA3 αE2 |3iA |2iE |1i1 |3i2 + αA3 αE3 |3iA |3iE |0i1 |1i2 .

(35)

Now, we need to apply iterative interactions in the register-agent subspace to update the agent in each step until we get maximal
learning fidelity with respect to the environment. We start by performing a GXOR gate between the register R1 and the agent,
|Ψi5

GXOR
= U(R
|Ψi4 ,
1 ,A)

|Ψi5

= αA0 αE0 |0iA |0iE |0i1 |0i2 + αA0 αE1 |3iA |1iE |3i1 |2i2 + αA0 αE2 |2iA |2iE |2i1 |0i2 + αA0 αE3 |1iA |3iE |1i1 |2i2
+αA1 αE0 |0iA |0iE |1i1 |1i2 + αA1 αE1 |3iA |1iE |0i1 |3i2 + αA1 αE2 |2iA |2iE |3i1 |1i2 + αA1 αE3 |1iA |3iE |2i1 |3i2
+αA2 αE0 |0iA |0iE |2i1 |2i2 + αA2 αE1 |3iA |1iE |1i1 |0i2 + αA2 αE2 |2iA |2iE |0i1 |2i2 + αA2 αE3 |1iA |3iE |3i1 |0i2
+αA3 αE0 |0iA |0iE |3i1 |3i2 + αA3 αE1 |3iA |1iE |2i1 |1i2 + αA3 αE2 |2iA |2iE |1i1 |3i2 + αA3 αE3 |1iA |3iE |0i1 |1i2 .

(36)

Hereafter, we apply the GXOR gate in the R2 -agent subspace,
|Ψi6

GXOR
= U(R
|Ψi5 ,
2 ,A)

|Ψi6

= αA0 αE0 |0iA |0iE |0i1 |0i2 + αA0 αE1 |3iA |1iE |3i1 |2i2 + αA0 αE2 |2iA |2iE |2i1 |0i2 + αA0 αE3 |1iA |3iE |1i1 |2i2
+αA1 αE0 |1iA |0iE |1i1 |1i2 + αA1 αE1 |0iA |1iE |0i1 |3i2 + αA1 αE2 |3iA |2iE |3i1 |1i2 + αA1 αE3 |2iA |3iE |2i1 |3i2
+αA2 αE0 |2iA |0iE |2i1 |2i2 + αA2 αE1 |1iA |1iE |1i1 |0i2 + αA2 αE2 |0iA |2iE |0i1 |2i2 + αA2 αE3 |3iA |3iE |3i1 |0i2
+αA3 αE0 |3iA |0iE |3i1 |3i2 + αA3 αE1 |2iA |1iE |2i1 |1i2 + αA3 αE2 |1iA |2iE |1i1 |3i2 + αA3 αE3 |0iA |3iE |0i1 |1i2 .

(37)

Afterwards, we perform a GXOR gate between R1 and A,
|Ψi7

GXOR
= U(R
|Ψi6 ,
1 ,A)

|Ψi7

= αA0 αE0 |0iA |0iE |0i1 |0i2 + αA0 αE1 |0iA |1iE |3i1 |2i2 + αA0 αE2 |0iA |2iE |2i1 |0i2 + αA0 αE3 |0iA |3iE |1i1 |2i2
+αA1 αE0 |0iA |0iE |1i1 |1i2 + αA1 αE1 |0iA |1iE |0i1 |3i2 + αA1 αE2 |0iA |2iE |3i1 |1i2 + αA1 αE3 |0iA |3iE |2i1 |3i2
+αA2 αE0 |0iA |0iE |2i1 |2i2 + αA2 αE1 |0iA |1iE |1i1 |0i2 + αA2 αE2 |0iA |2iE |0i1 |2i2 + αA2 αE3 |0iA |3iE |3i1 |0i2
+αA3 αE0 |0iA |0iE |3i1 |3i2 + αA3 αE1 |0iA |1iE |2i1 |1i2 + αA3 αE2 |0iA |2iE |1i1 |3i2 + αA3 αE3 |0iA |3iE |0i1 |1i2 .

(38)
8/16

Subsequently, an interaction in the R2 -agent subspace is performed,
|Ψi8

GXOR
|Ψi7 ,
= U(R
2 ,A)

|Ψi8

= αA0 αE0 |0iA |0iE |0i1 |0i2 + αA0 αE1 |2iA |1iE |3i1 |2i2 + αA0 αE2 |0iA |2iE |2i1 |0i2 + αA0 αE3 |2iA |3iE |1i1 |2i2
+αA1 αE0 |1iA |0iE |1i1 |1i2 + αA1 αE1 |3iA |1iE |0i1 |3i2 + αA1 αE2 |1iA |2iE |3i1 |1i2 + αA1 αE3 |3iA |3iE |2i1 |3i2
+αA2 αE0 |2iA |0iE |2i1 |2i2 + αA2 αE1 |0iA |1iE |1i1 |0i2 + αA2 αE2 |2iA |2iE |0i1 |2i2 + αA2 αE3 |0iA |3iE |3i1 |0i2
+αA3 αE0 |3iA |0iE |3i1 |3i2 + αA3 αE1 |1iA |1iE |2i1 |1i2 + αA3 αE2 |3iA |2iE |1i1 |3i2 + αA3 αE3 |1iA |3iE |0i1 |1i2 .

(39)

Finally, we apply a GXOR gate between R1 and the agent,
|Ψi9

GXOR
|Ψi8 ,
= U(R
1 ,A)

|Ψi9

= αA0 αE0 |0iA |0iE |0i1 |0i2 + αA0 αE1 |1iA |1iE |3i1 |2i2 + αA0 αE2 |2iA |2iE |2i1 |0i2 + αA0 αE3 |3iA |3iE |1i1 |2i2
+αA1 αE0 |0iA |0iE |1i1 |1i2 + αA1 αE1 |1iA |1iE |0i1 |3i2 + αA1 αE2 |2iA |2iE |3i1 |1i2 + αA1 αE3 |3iA |3iE |2i1 |3i2
+αA2 αE0 |0iA |0iE |2i1 |2i2 + αA2 αE1 |1iA |1iE |1i1 |0i2 + αA2 αE2 |2iA |2iE |0i1 |2i2 + αA2 αE3 |3iA |3iE |3i1 |0i2
+αA3 αE0 |0iA |0iE |3i1 |3i2 + αA3 αE1 |1iA |1iE |2i1 |1i2 + αA3 αE2 |2iA |2iE |1i1 |3i2 + αA3 αE3 |3iA |3iE |0i1 |1i2 .

(40)

As we can see, based in the quantum protocol described previously (see Fig 3), we have shown that for a fixed number of
interactions, we obtain maximal learning fidelity even though the system has an arbitrary dimension.

Quantum Reinforcement Learning in Multiqudit Systems
In the previous section, we proved that for an agent and environment composed of a multilevel system each, the quantum
reinforcement learning protocol entails maximal learning fidelity for a fixed number of steps, irrespective of the dimension.
Here, using this result, we also prove that for more than one multilevel system in agent, environment, and register subspaces,
the number of steps is also fixed and scales with the number of individual subsystems that compose both agent and environment
subsystems. To be more specific, in the single-multilevel case the needed total steps are nine. For two multilevel systems, we
show that the number of required steps are eighteen, and in general, 9n, with n being the number of multilevel subsystems. The
possible initial states of our protocol consist in arbitrary superpositions for both agent and environment states and the register
states are in their ground state,
N−1 N−1

|Ψ0 i =

∑ ∑

αAnm αEpq |niA |miA |piE |qiE |0i1 |0i2 |0i3 |0i4 .

(41)

n,m=0 p,q=0

The first step in the protocol consists in encoding the environment information in the register states. This is done by applying a
pair of GXOR gates. The gates are applied in the environment-register subspace, while the interaction in this case is the same
as the one described previously. Namely, E1 controls R1 and E2 controls R2 .
GXOR GXOR
|Ψ1 i = U(E
U
|Ψ0 i,
2 ,R2 ) (E1 ,R1 )
N−1 N−1

|Ψ1 i =

∑ ∑

αAnm αEpq |niA |miA |piE |qiE |pi1 |qi2 |0i3 |0i4 .

(42)

n,m=0 p,q=0

Similarly, in the second step we encode the environment information in the other two registers (R3 and R4 ) through GXOR
gates. Here, the control system is the environment while the targets are the registers.
GXOR GXOR
|Ψ2 i = U(E
U
|Ψ1 i,
2 ,R4 ) (E1 ,R3 )
N−1 N−1

|Ψ2 i =

∑ ∑

αAnm αEpq |niA |miA |piE |qiE |pi1 |qi2 |pi3 |qi4 .

(43)

n,m=0 p,q=0

Subsequently, a part of the register subspace is updated conditional on the agent information. Therefore, we apply a pair of
GXOR gates on the agent-register subspace. In this case, agents A1 and A2 are controls and registers R1 and R2 targets.
GXOR GXOR
|Ψ3 i = U(A
U
|Ψ2 i,
2 ,R2 ) (A1 ,R1 )
N−1 N−1

|Ψ3 i =

∑ ∑

αAnm αEpq |niA |miA |piE |qiE |n

pi1 |m

qi2 |pi3 |qi4 .

(44)

n,m=0 p,q=0

9/16

A
E
R1
R2
Figure 4. Quantum reinforcement learning for larger environment systems. The systems involved are denoted as agent A,
environment E and registers R1 , R2 , where E contains now two qubits while A just one. The logical gates applied between the
different subsystems are CNOT gates. In this case, to obtain maximal learning fidelity, it is required to perform two separate
measurements denoted by the blue boxes.
Now, we update the register subspace considering interactions between register components which have been acted upon with
the same part of the environment. Namely, the register R3 will be updated with the control of R1 (Similarly with R4 being
controlled with R2 ).
GXOR GXOR
|Ψ4 i = U(R
U
|Ψ3 i,
2 ,R4 ) (R1 ,R3 )
N−1 N−1

|Ψ4 i =

∑ ∑

αAnm αEpq |niA |miA |piE |qiE |n

pi1 |m

qi2 |n

2pi3 |m

2qi4 .

(45)

n,m=0 p,q=0

Subsequently, we need to apply successive interactions between agent states and register states to obtain maximal learning
fidelity. We show that applying the same interactions as for the single multilevel case for the triplet formed by agent A1 with the
environment parts R1 and R3 (similarly A2 with R2 and R4 ), the maximal learning fidelity is reached. It is straightforward to
show that
GXOR GXOR GXOR GXOR GXOR GXOR GXOR GXOR GXOR GXOR
U
U
U
U
U
U
|Ψ4 i,
|Ψ9 i = U(R
U
U
U
2 ,A2 ) (R1 ,A1 ) (R4 ,A2 ) (R3 ,A1 ) (R2 ,A2 ) (R1 ,A1 ) (R4 ,A2 ) (R3 ,A1 ) (R2 ,A2 ) (R1 ,A1 )
N−1 N−1

|Ψ9 i =

∑ ∑

αAnm αEpq |piA |qiA |piE |qiE |n

pi1 |m

qi2 |n

2pi3 |m

2qi4 .

(46)

n,m=0 p,q=0

Summarizing, for the case studied in this section, we demonstrate that the number of operations required to obtain maximal
learning fidelity does not depend on the learning unit dimension and it is equal to eighteen operations, which correspond to the
double of the required steps in the single multiqubit case. It is straightforward to realize that the number of needed operations
to achieve maximal learning fidelity in a machine learning protocol composed by n subsystems for agent and environment is
equal to 9n. Namely, the number of operations scales polynomially, indeed linearly, with the number of subsystems.

Quantum Reinforcement Learning in Larger Environments
Up to now, the quantum reinforcement learning protocol described here always considers that the agent and the environment
have the same number of subsystems, as well as the same dimension. In these cases, we have shown that by adding more system
registers the quantum protocol improves in the sense that only one iteration and one measurement is enough to obtain maximal
learning fidelity. Nevertheless, in more realistic scenarios, the agent must adapt to larger or more complex surroundings. Here,
we discuss the situation where the environment has more subsystems than the agent, and therefore a larger dimension. As the
environment has more information than the agent, it is expect that not all available surrounding information will be transferred
to the agent. Indeed, we prove that by depending on the register-environment interaction, the agent can encode the information
from one specific part of the environment. In this case, unlike the protocol previously discussed, we achieve maximal learning
fidelity after applying one measurement and a rewarding iteration.

10/16

The proposed quantum protocol is shown in Fig. 4. Here, one two-level system forms the agent, while register and
environment are constituted each by two qubits. Each environment qubit interacts with one qubit from the register, such that
this interaction updates the registers conditionally to the environment information. Then, one part of the register subspace is
also upgraded conditionally to the agent state. Subsequently, we perform a measurement on the register subspace, such that
depending on the measurement outcomes we apply a conditional operation in the agent-register subspace until the agent adapts
to a specific part of the environment. To illustrate this, let us introduce a possible agent-register-subspace state which has the
following form,
{|Ai = αA0 |0iA + αA1 |1iA , |Ei = αE00 |00iE + αE01 |01iE + αE10 |10iE + αE11 |11iE , |Ri = |0i1 |0i2 },

(47)

|Ψi0 = |Ai|Ei|Ri.

(48)

The first step is to transfer quantum information from the environment onto the registers. This is done by applying a pair of
CNOT gates in the environment-register subspaces,
|Ψi1

CNOT CNOT
= U(E,R
U
|Ψi0 ,
2 ) (E,R1 )

|Ψi1

= (αA0 |0iA + αA1 |1iA )(αE00 |00iE |0i1 |0i2 + αE01 |01iE |0i1 |1i2 + αE10 |10iE |1i1 |0i2 + αE11 |11iE |1i1 |1i1 ).

(49)

Subsequently, the register R1 is updated conditionally to the agent information. Therefore, a CNOT gate is applied in the
agent-register subspace, where the agent qubit is the control and the register R1 is the target,
|Ψi2

CNOT
= U(A,R
|Ψi1 ,
1)

|Ψi2

= αA0 αE00 |0iA |00iE |0i1 |0i2 + αA0 αE01 |0iA |01iE |0i1 |1i2 + αA0 αE10 |0iA |10iE |1i1 |0i2 + αA0 αE11 |0iA |11iE |1i1 |1i1
+αA1 αE00 |1iA |00iE |1i1 |0i2 + αA1 αE01 |1iA |01iE |1i1 |1i2 + αA1 αE10 |1iA |10iE |0i1 |0i2 + αA1 αE11 |1iA |11iE |0i1 |1i1 .

Afterwards, we perform a measurement on the register subspace. In this case, the wave function is projected into the four
possible measurement outcomes,
M1 = (αA0 αE00 |0iA |00iE + αA1 αE10 |1iA |10iE )|0i1 |0i2 = (αA0 αE00 |0iA |0iE1 + αA1 αE10 |1iA |1iE1 )|0iE2 |0i1 |0i2 ,
M2 = (αA0 αE01 |0iA |01iE + αA1 αE11 |1iA |11iE )|0i1 |1i2 = (αA0 αE01 |0iA |0iE1 + αA1 αE11 |1iA |1iE1 )|1iE2 |0i1 |1i2 ,
M3 = (αA1 αE00 |1iA |00iE + αA0 αE10 |0iA |10iE )|1i1 |0i2 = (αA1 αE00 |1iA |0iE1 + αA0 αE10 |0iA |1iE1 )|0iE2 |1i1 |0i2 ,
M4 = (αA0 αE11 |0iA |11iE + αA1 αE01 |1iA |01iE )|1i1 |1i1 = (αA0 αE11 |0iA |1iE1 + αA1 αE01 |1iA |0iE1 )|1iE2 |1i1 |1i1 .
As we can see, the projective measurement on the register subspace produces that agent and one part of the environment
subspace (E1 ) is in an entangled state. At this stage, we can apply the rewarding criterion which consists in performing a CNOT
gate operation in the register-agent subspace. The register qubit R1 is the control and the agent is the target,
0 00
1 10
M1a = UCNOT
(R1 ,A) M1 = (αA αE |0iA |0iE1 + αA αE |1iA |1iE1 )|0iE2 |0i1 |0i2 ,
0 01
1 11
M2a = UCNOT
(R1 ,A) M2 = (αA αE |0iA |0iE1 + αA αE |1iA |1iE1 )|1iE2 |0i1 |1i2 ,
1 00
0 10
M3a = UCNOT
(R1 ,A) M3 = (αA αE |0iA |0iE1 + αA αE |1iA |1iE1 )|0iE2 |1i1 |0i2 ,
0 11
1 01
M4a = UCNOT
(R1 ,A) M4 = (αA αE |1iA |1iE1 + αA αE |0iA |0iE1 )|1iE2 |1i1 |1i1 .

(50)

Finally, we perform a CNOT gate in the agent-register subspace to obtain orthogonal measurement outcomes. The qubit agent
is the control and the qubit register R1 is the target, according to
0 00
1 10
M1b = UCNOT
(A,R1 ) M1a = αA αE |0iA |00iE |0i1 |0i2 + αA αE |1iA |10iE |1i1 |0i2 ,
0 01
1 11
M2b = UCNOT
(A,R1 ) M2a = αA αE |0iA |01iE |0i1 |1i2 + αA αE |1iA |11iE |1i1 |1i2 ,
1 00
0 10
M3b = UCNOT
(A,R1 ) M3a = αA αE |0iA |00iE |1i1 |0i2 + αA αE |1iA |10iE |0i1 |0i2 ,
1 01
0 11
M4b = UCNOT
(A,R1 ) M4a = αA αE |0iA |01iE |1i1 |1i2 + αA αE |1iA |11iE |0i1 |1i1 .

(51)

In this quantum reinforcement learning protocol, we perform interactions between the environment and the register subspaces.
Nevertheless, the agent is updated only regarding the information encoded in register R1 . Thus, the maximal learning fidelity is
achieved with respect to the first qubit of the environment.
Let us now consider another configuration similar to the one studied previously in this article, where the register is formed
by a larger number of subsystems than the environment. Here, additionally, the environment we consider is larger than the agent.
11/16

We prove that, for this system configuration, maximal learning fidelity between the agent and one part of the environment is
achieved in one rewarding process. For this configuration, the maximal fidelity does not depend on the entanglement present in
the agent-environment subspace. The general agent-register-environment state is
|Ai = αA0 |0iA + αA1 |1iA ,
|Ei = (αE0 |0iE1

+ αE1 |1iE1 )|0iE2

(52)
+ (βE0 |0iE1

+ βE1 |1iE1 )|1iE2 ,

(53)

|Ri = |0i1 |0i2 |0i3 |0i4 ,

(54)

|Ψi0 = |Ai|Ei|Ri.

(55)

The quantum protocol consists in updating the registers R1,2 conditionally to the environment state E1,2 ,
|Ψi1

CNOT CNOT
= U(E
U
|Ψi0 ,
2 ,R2 ) (E1 ,R1 )

|Ψi1

= (αA0 |0iA + αA1 |1iA )(αE0 |0iE1 |0iE2 |0i1 |0i2 |0i3 |0i4 + αE1 |1iE1 |0iE2 |1i1 |0i2 |0i3 |0i4
+βE0 |0iE1 |1iE2 |0i1 |1i2 |0i3 |0i4 + βE1 |1iE1 |1iE2 |1i1 |1i2 |0i3 |0i4 ).

(56)

After this, we also update the information of the registers R3,4 conditionally to the environment state E1,2 ,
|Ψi2

CNOT CNOT
= U(E
U
|Ψi1 ,
2 ,R4 ) (E1 ,R3 )

|Ψi2

= (αA0 |0iA + αA1 |1iA )(αE0 |0iE1 |0iE2 |0i1 |0i2 |0i3 |0i4 + αE1 |1iE1 |0iE2 |1i1 |0i2 |1i3 |0i4
+βE0 |0iE1 |1iE2 |0i1 |1i2 |0i3 |1i4 + βE1 |1iE1 |1iE2 |1i1 |1i2 |1i3 |1i4 ).

(57)

Now, the register R1 is updated conditionally to the agent state,
|Ψi3

CNOT
= U(A,R
|Ψi2 ,
1)

|Ψi3

= αA0 αE0 |0iA |0iE1 |0iE2 |0i1 |0i2 |0i3 |0i4 + αA0 αE1 |0iA |1iE1 |0iE2 |1i1 |0i2 |1i3 |0i4 + αA0 βE0 |0iA |0iE1 |1iE2 |0i1 |1i2 |0i3 |1i4
+αA0 βE1 |0iA |1iE1 |1iE2 |1i1 |1i2 |1i3 |1i4 + αA1 αE0 |1iA |0iE1 |0iE2 |1i1 |0i2 |0i3 |0i4 + αA1 αE1 |1iA |1iE1 |0iE2 |0i1 |0i2 |1i3 |0i4
+αA1 βE0 |1iA |0iE1 |1iE2 |1i1 |1i2 |0i3 |1i4 + αA1 βE1 |1iA |1iE1 |1iE2 |0i1 |1i2 |1i3 |1i4 .

(58)

Then, the next step would consist in updating a part of the register subspace from the information encoded in the other part.
However, this step is not necessary because the number of terms in Eq. (58) is smaller than all the possible measurement
outcomes in the register subspace. Thus, the register is always projected onto orthogonal measurement outcomes. On the other
hand, we update the agent state from the information encoding in the register R1 . Therefore, we perform a CNOT gate in the
register-agent subspace, where the register R1 is the control and the agent is the target,
|Ψi4

CNOT
= U(R
|Ψi3 ,
1 ,A)

|Ψi4

= αA0 αE0 |0iA |0iE1 |0iE2 |0i1 |0i2 |0i3 |0i4 + αA0 αE1 |1iA |1iE1 |0iE2 |1i1 |0i2 |1i3 |0i4 + αA0 βE0 |0iA |0iE1 |1iE2 |0i1 |1i2 |0i3 |1i4
+αA0 βE1 |1iA |1iE1 |1iE2 |1i1 |1i2 |1i3 |1i4 + αA1 αE0 |0iA |0iE1 |0iE2 |1i1 |0i2 |0i3 |0i4 + αA1 αE1 |1iA |1iE1 |0iE2 |0i1 |0i2 |1i3 |0i4
+αA1 βE0 |0iA |0iE1 |1iE2 |1i1 |1i2 |0i3 |1i4 + αA1 βE1 |1iA |1iE1 |1iE2 |0i1 |1i2 |1i3 |1i4 .

(59)

By measuring the register subspace, we obtain that agent and environment qubit E1 achieve maximal fidelity.

Quantum Reinforcement Learning for Mixed States
Let us consider now the situation where the environment evolves under a noisy mechanism (for qubit states, noisy mechanisms
can be depolarizing noise as well as amplitude damping). In this case, the density matrix describing the environment state reads


ρ00 ρ01
.
(60)
ρ=
∗
ρ01
ρ11
We focus now our attention in the application of the quantum reinforcement learning protocol in this type of state. We will
show that, by adding more registers, two main results will be obtained. Firstly, even though the environment is in a mixed
state, the learning fidelity will be maximal for any measurement outcome in the register basis. Additionally, the measurement
outcomes provide relevant information about the coherences of the mixed state. To apply the quantum protocol, we express the

12/16

mixed state in term of its (non-unique) purification, such as


s

√
ρ10
|ρ10 |2
|ΨE+e i
=
ρ00 |0i + √ |1i |e1 i +
ρ11 −
|1i|e2 i,
ρ00
ρ00
s

√
√
|ρ10 |2
ρ10
ρ11 −
|e2 i → |ΨE+e i = ρ00 |0i|e1 i + ρ11 |1i|ψ̄e i.
|ψe i
= √ |e1 i +
ρ00
ρ00

(61)

(62)

Here, |ψ̄e i is a normalized vector in the purification Hilbert space. As we can see, the coefficient of the quantum state written in
its extended Hilbert space (environment + purification) depends only on the diagonal terms of the mixed state. Moreover, to
obtain additional information about the mixed state, we need to perform unitary transformations on it in such a way that the
information related to the coherences is in the diagonal of the state after the transformation. To be more specific, we need to
perform unitary transformations such that the mixed state can be written as follows,


∗ )
∗ )
1
1 + (ρ01 + ρ01
ρ11 − ρ00 + (ρ01 − ρ01
†
ρ̄ → Uy ρUy =
,
(63)
∗ )
∗ )
1 − (ρ01 + ρ01
2 ρ11 − ρ00 − (ρ01 − ρ01
ρ̃

→ Ux ρUx†



∗ )
∗ + i(ρ − ρ )
1
1 − i(ρ01 − ρ01
ρ01 + ρ01
11
00
.
=
∗ − i(ρ − ρ )
∗ )
1 + i(ρ01 − ρ01
2 ρ01 + ρ01
11
00

(64)

To carry out this task, we need to add three more registers, where each of them has the function to encode information of
diagonal, real, and imaginary part of the coherence terms, respectively. A possible state for the space composed of agent, mixed
environment and register is given by
{|Ai = αA0 |0iA + αA1 |1iA , |ΨE+e i =

√
√
1
ρ00 |0i|e1 i + ρ11 |1i|ψ̄e i, |Ri = |0i1 |0i2 √ (|1i3 |0i4 |0i5 + |0i3 |1i4 |0i5
3

+|0i3 |0i4 |1i5 )}
|Ψi0 = |Ai|ΨE+e i|Ri.

(65)

The first step is to apply a unitary transformation in the environment state according to
|Ψi1

= |Ai|ψE+e i|0i1 |0i2 |1i3 |0i4 |0i5 + |AiUy |ψE+e i|0i1 |0i2 |0i3 |1i4 |0i5 + |AiUx |ψE+e i|0i1 |0i2 |0i3 |0i4 |1i5 ,
√
√
1
|Ψ1 i = √ (αA0 |0iA + αA1 |1iA ) ( ρ00 |0i|e1 i + ρ11 |1i|ψ̄e i)|0i1 |0i2 |1i3 |0i4 |0i5
3
r
r
1
1
+ Re(ρ01 )|0i|e1 i +
− Re(ρ01 )|1i|ψ̄e i)|0i1 |0i2 |0i3 |1i4 |0i5
+(
2
2
r
r
1
1
+(
+ Im(ρ01 )|0i|e1 i +
− Im(ρ01 )|1i|ψ̄e i)|0i1 |0i2 |0i3 |0i4 |1i5 ].
(66)
2
2

Afterwards, we apply the quantum protocol as we did in the first section. Namely, we first update the register conditionally to
the information of the environment. Then, we update the register R1 conditionally to the information of the agent. Subsequently,
to obtain orthogonal measurement outcomes we perform CNOT gates in the register subspace (R1 is the control and R2 is the
agent). Finally, the agent is updated in terms of the information encoded in register R1 (where A is the target and R1 is the
control),
|Ψi5

=

√
√
1
√ (αA0 ρ00 |0iA |0iE |e1 i|0i1 |0i2 |1i3 |0i4 |0i5 + αA0 ρ11 |1iA |1iE |ψ̄e i|1i1 |0i2 |1i3 |0i4 |0i5
3
√
√
+αA1 ρ00 |0iA |0iE |e1 i|1i1 |1i2 |1i3 |0i4 |0i5 + αA1 ρ11 |0iA |0iE |ψ̄e i|0i1 |1i2 |1i3 |0i4 |0i5
r
r
1
1
0
0
+αA
+ Re(ρ01 )|0iA |0iE |e1 i|0i1 |0i2 |0i3 |1i4 |0i5 + αA
− Re(ρ01 )|1iA |1iE |ψ̄e i|1i1 |0i2 |0i3 |1i4 |0i5
2
2
r
r
1
1
+αA1
+ Re(ρ01 )|0iA |0iE |e1 i|1i1 |1i2 |0i3 |1i4 |0i5 + αA1
− Re(ρ01 )|0iA |0iE |ψ̄e i|0i1 |1i2 |0i3 |1i4 |0i5
2
2
r
r
1
1
+ Im(ρ01 )|0iA |0iE |e1 i|0i1 |0i2 |0i3 |1i4 |0i5 + αA0
− Im(ρ01 )|1iA |1iE |ψ̄e i|1i1 |0i2 |0i3 |1i4 |0i5
+αA0
2
2
r
r
1
1
+αA1
+ Im(ρ01 )|0iA |0iE |e1 i|1i1 |1i2 |0i3 |1i4 |0i5 + αA1
− Im(ρ01 )|0iA |0iE |ψ̄e i|0i1 |1i2 |0i3 |1i4 |0i5 ).
2
2
13/16

This quantum reinforcement learning protocol shows two interesting features. First, we recover the same result as the one
studied in the first section, i.e., the learning fidelity is maximal independently of the measurement outcomes in the register
subspace. The second feature is that, for specific measurement outcomes in a part of the register subspace, we obtain information
about the population (diagonal) and the coherences (non-diagonals) of the mixed state. This feature can be used in problems
such as partial cloning in cases where the system in which we can extract information evolves under loss mechanisms.

Analysis of Implementation in Quantum Technologies
An interesting result obtained in this manuscript is that in most of the cases, for the considered quantum reinforcement learning
protocols, adding more registers improves the rewarding process. That is, via a purely unitary evolution, without coherent
feedback, a maximally positively-correlated agent environment state is achieved, in the sense that the final agent contains the
same quantum information as the considered final environment. This means that the agent has acquired the needed information
about the environment and accordingly modified it, being this a quantum process. In our formalism, typically, one measurement
at the end of the protocol is enough to obtain maximal learning fidelity in one iteration of the process. In this sense, several
quantum architectures could benefit of this fact, given that coherent feedback is not needed in this case. For instance, we focus
our attention in two prominent platforms, namely, trapped ions and superconducting circuits.
Trapped Ions
As we have pointed out along the manuscript, the performance of our proposed quantum protocols is based on the quality of
the quantum gates between different subsystems. In this case, the realization of high-fidelity quantum gates is essential to
perform the quantum protocol proposed here. Technological progress in trapped ions has enabled to implement single 44 and
two-qubit quantum gates45 with a large fidelity. For the single-qubit gate, e.g., a Beryllium hyperfine transition can be driven
with microwave fields or lasers, being the error associated with single-qubit gates below 10−4 . For two-qubit gates, the use of
either microwaves or a laser beam with modulated amplitude allows for the interaction of both qubits (electronic levels of, e.g.,
Beryllium or Calcium ions) at the same time. Adiabatic elimination of the motion allows one to obtain maximally entangled
states of both ions. The fidelity of trapped-ion two-qubit gates can reach nowadays above 99.9%46, 47 . Trapped-ion technologies
offer long coherences times, which can reach up to the range of seconds48 for Calcium atoms. In addition, this platform enables
state preparation and readout with high fidelity34, 49, 50 . Here, the use of hyperfine states and the microwave fields improve the
optical pumping fidelity and improve the relaxation time T1 allowing to obtain fidelity readouts of 99.9999%49 .
Superconducting Circuits
As in trapped ions, the technological progress in superconducting circuits has grown significantly in the latter years. For instance,
artificial atoms whose coherence times are in the microsecond range have been built in coplanar38 and 3D architectures39 . On
the other hand, integrated Josephson quantum processors allows one to implement quantum gates between two-level systems
even in cases where the qubits do not have identical frequencies, as well as making them interact via a quantum bus51 . The
Xmon qubits achieve two-qubit gate fidelities above 99%36, 37 . These technological progresses have developed feedback loop
control in this platform. This feedback protocol relies on high fidelity readout, as well as on conditional control on the outcome
of a quantum non-demolition measurement40, 41 . Even though in the quantum reinforcement learning protocols in this paper
coherent feedback is not required, this may be a useful ingredient in other quantum reinforcement learning proposals21 .

Discussion
In summary, we propose a protocol to perform quantum reinforcement learning which does not require coherent feedback and,
therefore, may be implemented in a variety of quantum technologies. Adding more registers than in previous proposals in
the literature21 , the rewarding criterion can be applied at the end of the protocol, while agent and environment need not be
measured directly, although only via the registers. We also obtain that when the considered systems are composed of qudits,
the number of steps needed to obtain maximal learning fidelity is fixed in each qudit dimension and scales polynomially with
the number of qudit subsystems. We consider as well environment states which are mixtures, while the agent can also in this
case acquire the appropriate information from them. Implementations of these protocols in trapped ions and superconducting
circuits seem feasible with current platforms.

References
1. Michalski, R. S., Carbonell, J. G. & Mitchell, T. M. Machine learning: An artificial intelligence approach (Springer
Science & Business Media, 2013).
2. Plamondon, R. & Srihari, S. N. Online and off-line handwriting recognition: a comprehensive survey. IEEE Transactions
on Pattern Analysis Mach. Intell. 22, 63–84 (2000).
14/16

3. Lee, K. F., Hon, H. W., Hwang, M. Y., Mahajan, S. & Reddy, R. The sphinx speech recognition system. In International
Conference on Acoustics, Speech, and Signal Processing,, 445–448 vol.1 (1989).
4. Silver, D. et al. Mastering the game of Go with deep neural networks and tree search, Nature 529, 484–489 (2016).
5. Russell, S. J. & Norvig, P. Artificial Intelligence: A Modern Approach (International Edition) (Pearson US Imports &
PHIPEs, 2002).
6. Sutton, R. S. & Barto, A. G. Reinforcement learning: An introduction, vol. 1 (MIT press, Cambridge, USA, 1998).
7. Wittek, P. Quantum machine learning: what quantum computing means to data mining (Academic Press, 2014).
8. Schuld, M., Sinayskiy, I. & Petruccione, F. An introduction to quantum machine learning. Contemp. Phys. 56, 172–185
(2015).
9. Adcock, J. et al. Advances in quantum machine learning. arXiv:1512.02900 (2015).
10. Biamonte, J. et al. Quantum machine learning. Nature 549, 195–202 (2017).
11. Dunjko, V. & Briegel, H. J. Machine learning & artificial intelligence in the quantum domain. arXiv:1709.02779 (2017).
12. Bonner, R. & Freivalds, R. A survey of quantum learning. Quantum Comput. Learn. 106 (2003).
13. Aı̈meur, E., Brassard, G. & Gambs, S. Quantum speed-up for unsupervised learning. Mach. Learn. 90, 261–287 (2013).
14. Lloyd, S., Mohseni, M. & Rebentrost, P. Quantum algorithms for supervised and unsupervised machine learning.
arXiv:1307.0411 (2013).
15. Rebentrost, P., Mohseni, M. & Lloyd, S. Quantum support vector machine for big data classification. Phys. Rev. Lett. 113,
130503 (2014).
16. Alvarez-Rodriguez, U., Lamata, L., Escandell-Montero, P., Martı́n-Guerrero, J. D., and Solano, E. Quantum Machine
Learning without Measurements. Sci. Rep., in press. arXiv:1612.05535.
17. Dong, D., Chen, C., Li, H. & Tarn, T. J. Quantum reinforcement learning. IEEE Transactions on Syst. Man, Cybern. Part
B (Cybernetics) 38, 1207–1220 (2008).
18. Paparo, G. D., Dunjko, V., Makmal, A., Martin-Delgado, M. A. & Briegel, H. J. Quantum speedup for active learning
agents. Phys. Rev. X 4, 031002 (2014).
19. Dunjko, V., Taylor, J. M. & Briegel, H. J. Quantum-enhanced machine learning. Phys. Rev. Lett. 117, 130501 (2016).
20. Crawford, D., Levit, A., Ghadermarzy, N., Oberoi, J. S. & Ronagh, P. Reinforcement learning using quantum boltzmann
machines. arXiv:1612.05695 (2016).
21. Lamata, L. Basic protocols in quantum reinforcement learning with superconducting circuits. Sci. Rep. 7, 1609 (2017).
22. Pfeiffer, P., Egusquiza, I. L., Di Ventra, M., Sanz, M., & Solano, E. Quantum Memristors, Sci. Rep. 6, 29507 (2016).
23. Salmilehto, J., Deppe, F., Di Ventra, M., Sanz, M., & Solano, E. Quantum Memristors with Superconducting Circuits. Sci.
Rep. 7, 42044 (2017).
24. Sanz, M., Lamata, L., & Solano, E. Quantum Memristors in Optical Systems with Feedback, submitted for publication.
25. Shevchenko, S. N., Pershin, Y. V., & Nori, F. Qubit-Based Memcapacitors and Meminductors, Phys. Rev. Applied 6, 014006
(2016).
26. Benedetti, M., Realpe-Gómez, J., & Perdomo-Ortiz, A. Quantum-assisted Helmholtz machines: A quantum-classical deep
learning framework for industrial datasets in near-term devices, arXiv:1708.09784.
27. Benedetti, M., Realpe-Gómez, J., Biswas, R., & Perdomo-Ortiz, A. Estimation of effective temperatures in quantum
annealers for sampling applications: A case study with possible applications in deep learning, Phys. Rev. A 94, 022308
(2016).
28. Perdomo-Ortiz, A., Benedetti, M., Realpe-Gómez, J., & Biswas, R. Opportunities and challenges for quantum-assisted
machine learning in near-term quantum computers, arXiv:1708.09757.
29. Leibfried, D., Blatt, R., Monroe, C. & Wineland, D. Quantum dynamics of single trapped ions. Rev. Mod. Phys. 75,
281–324 (2003).
30. Haffner, H., Roos, C. & Blatt, R. Quantum computing with trapped ions. Phys. Rep. 469, 155 – 203 (2008).
31. Blais, A. et al. Quantum-information processing with circuit quantum electrodynamics. Phys. Rev. A 75, 032329 (2007).
32. Clarke, J. & Wilhelm, F. K. Superconducting quantum bits. Nature 453, 1031–1042 (2008).
15/16

33. Wendin, G. Quantum information processing with superconducting circuits: a review. Rep. Prog. Phys. 80, 106001 (2017).
34. Harty, T. P. et al. High-fidelity preparation, gates, memory, and readout of a trapped-ion quantum bit. Phys. Rev. Lett. 113,
220501 (2014).
35. Ballance, C. J., Harty, T. P., Linke, N. M., Sepiol, M. A. & Lucas, D. M. High-fidelity quantum logic gates using trapped-ion
hyperfine qubits. Phys. Rev. Lett. 117, 060504 (2016).
36. Barends, R. et al. Superconducting quantum circuits at the surface code threshold for fault tolerance. Nature 508, 500–503
(2014).
37. Barends, R. et al. Digitized adiabatic quantum computing with a superconducting circuit. Nature 534, 222–226 (2016).
38. Barends, R. et al. Coherent josephson qubit suitable for scalable quantum integrated circuits. Phys. Rev. Lett. 111, 080502
(2013).
39. Paik, H. et al. Observation of high coherence in josephson junction qubits measured in a three-dimensional circuit qed
architecture. Phys. Rev. Lett. 107, 240501 (2011).
40. Ristè, D., Bultink, C. C., Lehnert, K. W. & DiCarlo, L. Feedback control of a solid-state qubit using high-fidelity projective
measurement. Phys. Rev. Lett. 109, 240502 (2012).
41. Ristè, D. & DiCarlo, L. Digital feedback in superconducting quantum circuits. arXiv:1508.01385 (2015).
42. Koch, J. et al. Charge-insensitive qubit design derived from the cooper pair box. Phys. Rev. A 76, 042319 (2007).
43. Alber, G., Delgado, A., Gisin, N. & Jex, I. Generalized quantum xor-gate for quantum teleportation and state purification
in arbitrary dimensional hilbert spaces. quant-ph/0008022 (2000).
44. Brown, K. R. et al. Single-qubit-gate error below 10−4 in a trapped ion. Phys. Rev. A 84, 030303 (2011).
45. Benhelm, J., Kirchmair, G., Roos, C. F. & Blatt, R. Towards fault-tolerant quantum computing with trapped ions. Nat.
Phys. 4, 463–466 (2008).
+

46. Gaebler, J. P. et al. High-fidelity universal gate set for 9 Be ion qubits. Phys. Rev. Lett. 117, 060505 (2016).
47. Harty, T. P. et al. High-fidelity trapped-ion quantum logic using near-field microwaves. Phys. Rev. Lett. 117, 140501
(2016).
48. Langer, C. et al. Long-lived qubit memory using atomic ions. Phys. Rev. Lett. 95, 060502 (2005).
49. Myerson, A. H. et al. High-fidelity readout of trapped-ion qubits. Phys. Rev. Lett. 100, 200502 (2008).
50. Noek, R. et al. High speed, high fidelity detection of an atomic hyperfine qubit. Opt. Lett. 38, 4735–4738 (2013).
51. Blais, A., Huang, R.-S., Wallraff, A., Girvin, S. M. & Schoelkopf, R. J. Cavity quantum electrodynamics for superconducting electrical circuits: An architecture for quantum computation. Phys. Rev. A 69, 062320 (2004).

Acknowledgements
F.A.C.-L. acknowledges support from CEDENNA basal grant No. FB0807 and Dirección de Postgrado USACH. J. C. R.
acknowledges the support from FONDECYT under grant No. 1140194. L. L. and E. S. acknowledge support from Spanish
MINECO/FEDER FIS2015-69983-P and Basque Government IT986-16, and L. L. from Ramón y Cajal Grant RYC-2012-11391.

Author Contributions
F. A. C.-L. developed the protocol under the guidance of J. C. R., based on original ideas and feedback by L. L. All authors
contributed to the development of the ideas, performance of the protocol and writing of the manuscript. E. S. supervised the
project in all stages.

Additional Information
Competing financial interests: The authors declare no competing financial interests.

16/16

