Feudal Reinforcement Learning for Dialogue Management in Large
Domains
Iñigo Casanueva1 , Paweł Budzianowski1 , Pei-Hao Su2 ,
Stefan Ultes1 , Lina Rojas-Barahona1 , Bo-Hsiang Tseng1 and Milica Gašić1
1
Department of Engineering, University of Cambridge, UK
2
PolyAI Limited, London, UK
ic340@cam.ac.uk

arXiv:1803.03232v1 [cs.CL] 8 Mar 2018

Abstract
Reinforcement learning (RL) is a promising
approach to solve dialogue policy optimisation. Traditional RL algorithms, however, fail
to scale to large domains due to the curse
of dimensionality. We propose a novel Dialogue Management architecture, based on Feudal RL, which decomposes the decision into
two steps; a first step where a master policy
selects a subset of primitive actions, and a second step where a primitive action is chosen
from the selected subset. The structural information included in the domain ontology is
used to abstract the dialogue state space, taking the decisions at each step using different
parts of the abstracted state. This, combined
with an information sharing mechanism between slots, increases the scalability to large
domains. We show that an implementation
of this approach, based on Deep-Q Networks,
significantly outperforms previous state of the
art in several dialogue domains and environments, without the need of any additional reward signal.

1

Introduction

Task-oriented Spoken Dialogue Systems (SDS),
in the form of personal assistants, have recently
gained much attention in both academia and industry. One of the most important modules of a
SDS is the Dialogue Manager (DM) (or policy),
the module in charge of deciding the next action in
each dialogue turn. Reinforcement Learning (RL)
(Sutton and Barto, 1999) has been studied for several years as a promising approach to model dialogue management (Levin et al., 1998; Henderson
et al., 2008; Pietquin et al., 2011; Young et al.,
2013; Casanueva et al., 2015; Su et al., 2016).
However, as the dialogue state space increases, the
number of possible trajectories needed to be explored grows exponentially, making traditional RL
methods not scalable to large domains.

Hierarchical RL (HRL), in the form of temporal abstraction, has been proposed in order to mitigate this problem (Cuayáhuitl et al., 2010, 2016;
Budzianowski et al., 2017; Peng et al., 2017).
However, proposed HRL methods require that the
task is defined in a hierarchical structure, which is
usually handcrafted. In addition, they usually require additional rewards for each subtask. Space
abstraction, instead, has been successfully applied
to dialogue tasks such as Dialogue State Tracking (DST) (Henderson et al., 2014b), and policy transfer between domains (Gašić et al., 2013,
2015; Wang et al., 2015). For DST, a set of binary classifiers can be defined for each slot, with
shared parameters, learning a general way to track
slots. The policy transfer method presented in
(Wang et al., 2015), named Domain Independent
Parametrisation (DIP), transforms the belief state
into a slot-dependent fixed size representation using a handcrafted feature function. This idea could
also be applied to large domains, since it can be
used to learn a general way to act in any slot.
In slot-filling dialogues, a HRL method that relies on space abstraction, such as Feudal RL (FRL)
(Dayan and Hinton, 1993), should allow RL scale
to domains with a large number of slots. FRL divides a task spatially rather than temporally, decomposing the decisions in several steps and using
different abstraction levels in each sub-decision.
This framework is especially useful in RL tasks
with large discrete action spaces, making it very
attractive for large domain dialogue management.
In this paper, we introduce a Feudal Dialogue
Policy which decomposes the decision in each turn
into two steps. In a first step, the policy decides if
it takes a slot independent or slot dependent action. Then, the state of each slot sub-policy is abstracted to account for features related to that slot,
and a primitive action is chosen from the previously selected subset. Our model does not require

any modification of the reward function and the
hierarchical architecture is fully specified by the
structured database representation of the system
(i.e. the ontology), requiring no additional design.

2

Main belief state (𝑏)

𝜙1 (𝑏)

𝜙* (𝑏)

𝑄,10

Qπ (b, a) = E{R|bt = b, at = a}
(1)
PT −1 (τ −t)
rτ and γ is a discount
where R =
τ =t γ
factor, with 0 ≤ γ ≤ 1.
The objective of RL is to find an optimal policy π ∗ , i.e. a policy that maximizes the expected
return in each belief state. In Value-based algorithms, the optimal policy can be found by greedily taking the action which maximises Qπ (b, a).
In slot-filling SDSs the belief state space B is
defined by the ontology, a structured representation of a database of entities that the user can retrieve by talking to the system. Each entity has
a set of properties, refereed to as slots S, where
each of the slots can take a value from the set Vs .
The belief state b is then defined as the concatenation of the probability distribution of each slot,
plus a set of general features (e.g. the communication function used by the user, the database
search method...) (Henderson et al., 2014a). The
set A is defined as a set of summary actions, where
the actions can be either slot dependent (e.g. request(food), confirm(area)...) or slot independent1
(e.g. hello(), inform()...).
The belief space B is defined by the ontology, therefore belief states of different domains
will have different shapes. In order to transfer
knowledge between domains, Domain Independent Parametrization (DIP) (Wang et al., 2015)
proposes to abstract the belief state b into a fixed
1
We include the summary actions dependent on all the
slots, such as inform(), in this group.

-

𝜋*

Argmax
over master
actions

𝑄,* - … 𝑄,* (

)

Argmax over slot
independent primitives

𝑄,10
/

𝑏")

𝜋2

"

…

𝜋"#

𝜋"(

"

… 𝜙") (𝑏)

𝑏"#

𝑏"(

𝜋1

𝑏*

Background

Dialogue management can be cast as a continuous MDP (Young et al., 2013) composed of a continuous multivariate belief state space B, a finite
set of actions A and a reward function R(bt , at ).
At a given time t, the agent observes the belief
state bt ∈ B, executes an action at ∈ A and receives a reward rt ∈ R drawn from R(bt , at ). The
action taken, a, is decided by the policy, defined
as the function π(b) = a. For any policy π and
b ∈ B, the Q-value function can be defined as the
expected (discounted) return R, starting from state
b, taking action a, and then following policy π until the end of the dialogue at time step T :

𝜙"( (𝑏) 𝜙"# (𝑏)

𝑏1

"

𝜋")

"

"

"

𝑄,(/ … 𝑄,(/ 𝑄,#/ … 𝑄,#/ … 𝑄,)/ … 𝑄,)/
(

)

(

)

(

)

Argmax over slots and slot
dependent primitives

Figure 1: Feudal dialogue architecture used in this
work. The sub-policies surrounded by the dashed line
have shared parameters. The simple lines show the data
flow and the double lines the sub-policy decisions.

size representation. As each action is either slot independent or dependent on a slot s, a feature function φdip (b, s) can be defined, where s ∈ S∪si and
si stands for slot independent actions. Therefore,
in order to compute the policy, Q(b, a) can be approximated as Q(φdip (b, s), a), where s is the slot
associated to action a.
Wang et al. (2015) presents a handcrafted feature function φdip (b, s). It includes the slot independent features of the belief state, a summarised
representation of the joint belief state, and a summarised representation of the belief state of the
slot s. Section 4 gives a more detailed description
of the φdip (b, s) function used in this work.

3

Feudal dialogue management

FRL decomposes the policy decision π(b) = a in
each turn into several sub-decisions, using different abstracted parts of the belief state in each subdecision. The objective of a task oriented SDS
is to fulfill the users goal, but as the goal is not
observable for the SDS, the SDS needs to gather
enough information to correctly fulfill it. Therefore, in each turn, the DM can decompose its decision in two steps: first, decide between taking
an action in order to gather information about the
user goal (information gathering actions) or taking an action to fulfill the user goal or a part of it
(information providing actions) and second, select
a (primitive) action to execute from the previously
selected subset. In a slot-filling dialogue, the set of
information gathering actions can be defined as the
set of slot dependent actions, while the set of information providing actions can be defined as the

remaining actions.
The architecture of the feudal policy proposed
by this work is represented schematically in Figure
1. The (primitive) actions are divided between two
subsets; slot independent actions Ai (e.g. hello(),
inform()); and slot dependent actions Ad (e.g. request(), confirm())2 . In addition, a set of master
m
m
actions Am = (am
i , ad ) is defined, where ai corresponds to taking an action from Ai and am
d to
taking an action from Ad . Then, a feature function
φs (b) = bs is defined for each slot s ∈ S, as well
as a slot independent feature function φi (b) = bi
and a master feature function φm (b) = bm . These
feature functions can be handcrafted (e.g. the DIP
feature function introduced in section 2) or any
function approximator can be used (e.g. neural
networks trained jointly with the policy).
Finally, a master policy πm (bm ) = am , a slot
independent policy πi (bi ) = ai and a set of slot
specific policies πs (bs ) = ad , one for each s ∈ S,
are defined, where am ∈ Am , ai ∈ Ai and
ad ∈ Ad . Contrary to other feudal policies, the
slot specific sub-policies have shared parameters,
in order to generalise between slots (following the
idea used by Henderson et al. (2014b) for DST).
The differences between the slots (size, value distribution...) are accounted by the feature function
φs (b). Therefore πm (bm ) is defined as:
πm (bm ) = argmax Qm (bm , am )

(2)

am ∈Am

If πm (bm ) = am
i , the sub-policy run is πi :
πi (bi ) = argmax Qi (bi , ai )

(3)

ai ∈Ai

Else, if πm (bm ) = am
d , πd is selected. This policy
runs each slot specific policy, πs , for all s ∈ S,
choosing the action-slot pair that maximises the Q
function over all the slot sub-policies.
πd (bs |∀s ∈ S) = argmax Qs (bs , ad )

(4)

ad ∈Ad ,s∈S

Then, the summary action a is constructed by joining ad and s (e.g. if ad =request() and s=food,
then the summary action will be request(food)). A
pseudo-code of the Feudal Dialogue Policy algorithm is given in Appendix A.

Domain
Cambridge Restaurants
San Francisco Restaurants
Laptops

SER
Masks
User

Env. 1
0%
on
Std.

Code
CR
SFR
LAP

Env. 2
0%
off
Std.

# constraint slots
3
6
11

Env. 3
15%
on
Std.

Env. 4
15%
off
Std.

# requests
9
11
21

# values
268
636
257

Env. 5
15%
on
Unf.

Env. 6
30%
on
Std.

Table 1: Sumarised description of the domains and
environments used in the experiments. Refer to
(Casanueva et al., 2017) for a detailed description.

4

Experimental setup

The models used in the experiments have been implemented using the PyDial toolkit (Ultes et al.,
2017)3 and evaluated on the PyDial benchmarking environment (Casanueva et al., 2017). This
environment presents a set of tasks which span
different size domains, different Semantic Error
Rates (SER), and different configurations of action masks and user model parameters (Standard
(Std.) or Unfriendly (Unf.)). Table 1 shows a summarised description of the tasks. The models developed in this paper are compared to the state-ofthe-art RL algorithms and to the handcrafted policy presented in the benchmarks.
4.1

DIP-DQN baseline

An implementation of DIP based on Deep-Q
Networks (DQN) (Mnih et al., 2013) is implemented as an additional baseline (Papangelis and
Stylianou, 2017). This policy, named DIP-DQN,
uses the same hyperparameters as the DQN implementation released in the PyDial benchmarks.
A DIP feature function based in the description in
(Wang et al., 2015) is used, φdip (b, s) = ψ0 (b) ⊕
ψj (b) ⊕ ψd (b, s), where:
• ψ0 (b) accounts for general features of the belief
state, such as the database search method.
• ψj (b) accounts for features of the joint belief
state, such as the entropy of the joint belief.
• ψd (b, s) accounts for features of the marginal
distribution of slot s, such as the entropy of s.
Appendix B shows a detailed description of the
DIP features used in this work.
4.2

Feudal DQN policy

A Feudal policy based on the architecture described in sec. 3 is implemented, named FDQN.
Each sub-policy is constructed by a DQN policy

2

Note that the actions of this set are composed just by the
communication function of the slot dependent actions, thus
reducing the number of actions compared to A.

3
The implementation of the models can be obtained in
www.pydial.org

Env. 1
Env. 2
Env. 3
Env. 4
Env. 5
Env. 6

Task
CR
SFR
LAP
CR
SFR
LAP
CR
SFR
LAP
CR
SFR
LAP
CR
SFR
LAP
CR
SFR
LAP

Feudal-DQN
Suc.
Rew.
89.3% 11.7
71.1%
7.1
65.5%
5.7
97.8% 13.1
95.4% 12.4
94.1% 12.0
92.6% 11.7
90.0%
9.7
89.6%
9.4
91.4% 11.2
90.3% 10.2
88.7%
9.8
96.3% 11.5
88.9%
7.9
78.8%
5.2
90.6% 10.4
83.0%
7.1
78.5%
6.0

DIP-DQN
Suc.
Rew.
48.8%
-2.8
25.8%
-7.4
26.6%
-8.8
85.5%
9.6
85.7%
8.4
89.5%
9.7
86.1%
8.9
59.3%
0.2
71.5%
3.1
82.6%
8.7
86.1%
9.2
74.8%
6.0
74.4%
2.9
75.5%
3.2
64.4%
-0.4
83.4%
8.1
71.9%
3.9
66.5%
2.7

Bnch.
Rew.
13.5
11.7
10.5
12.2
9.6
7.3
11.9
8.6
6.7
10.7
7.7
5.5
10.5
4.5
4.1
10.0
3.9
3.6

Hdc.
Rew.
14.0
12.4
11.7
14.0
12.4
11.7
11.0
9.0
8.7
11.0
9.0
8.7
9.3
6.0
5.3
9.7
6.4
5.5

Table 2: Success rate and reward for Feudal-DQN and
DIP-DQN in the 18 benchmarking tasks, compared
with the reward of the best performing algorithm in
each task (Bnch.) and the handcrafted policy (Hdc.)
presented in (Casanueva et al., 2017).

(Su et al., 2017). These policies have the same
hyperparameters as the baseline DQN implementation, except for the two hidden layer sizes, which
are reduced to 130 and 50 respectively. As feature
functions, subsets of the DIP features are used:
φm (b) = φi (b) = ψ0 (b) ⊕ ψj (b)
φs (b) = ψ0 (b) ⊕ ψj (b) ⊕ ψd (b, s) ∀s ∈ S
The original set of summary actions of the benchmarking environment, A, has a size of 5 + 3 ∗ |S|,
where |S| is the number of slots. This set is divided in two subsets4 : Ai of size 6 and Ad of size
4. Each sub-policy (including πm ) is trained with
the same sparse reward signal used in the baselines, getting a reward of 20 if the dialogue is successful or 0 otherwise, minus the dialogue length.

5

Results

The results in the 18 tasks of the benchmarking
environment after 4000 training dialogues are presented in Table 2. The same evaluation procedure of the benchmarks is used, presenting the
mean over 10 different random seeds and testing
every seed for 500 dialogues. The FDQN policy
substantially outperforms every other other policy in all the environments except Env. 1. The

Figure 2: Learning curves for Feudal-DQN and DIPDQN in Env. 4, compared to the two best performing
algorithms in (Casanueva et al., 2017) (DQN and GPSarsa). The shaded area depicts the mean ± the standard deviation over ten random seeds.

performance increase is more considerable in the
two largest domains (SFR and LAP), with gains
up to 5 points in accumulated reward in the most
challenging environments (e.g. Env. 4 LAP),
compared to the best benchmarked RL policies
(Bnch.). In addition, FDQN consistently outperforms the handcrafted policy (Hdc.) in environments 2 to 6, which traditional RL methods could
not achieve. In Env. 1, however, the results for
FDQN and DIP-DQN are rather low, specially for
DIP-DQN. Surprisingly, the results in Env. 2,
which only differs from Env. 1 in the absence of
action masks (thus, in principle, is a more complex environment), outperform every other algorithm. Analysing the dialogues individually, we
could observe that, in this environment, both policies are prone to “overfit” to an action 5 . The performance of FDQN and DIP-DQN in Env. 4 is also
better than in Env. 3, while the difference between
these environments also lies in the masks. This
suggests that an specific action mask design can
be helpful for some algorithms, but can harm the
performance of others. This is especially severe
in the DIP-DQN case, which shows good performance in some challenging environments, but it is
more unstable and prone to overfit than FDQN.
However, the main purpose of action masks is
to reduce the number of dialogues needed to train
a policy. Observing the learning curves shown
in Figure 2, the FDQN model can learn a nearoptimal policy in large domains in about 1500 dialogues, even if no additional reward is used, making the action masks unnecessary.

4

An additional pass() action is added to each subset,
which is taken whenever the other sub-policy is executed.
This simplifies the training algorithm.

5
The model overestimates the value of an incorrect action,
continuously repeating it until the user runs out of patience.

6

Conclusions and future work

We have presented a novel dialogue management
architecture, based on Feudal RL, which substantially outperforms the previous state of the art in
several dialogue environments. By defining a set
of slot dependent policies with shared parameters,
the model is able to learn a general way to act in
slots, increasing its scalability to large domains.
Unlike other HRL methods applied to dialogue,
no additional reward signals are needed and the hierarchical structure can be derived from a flat ontology, substantially reducing the design effort.
A promising approach would be to substitute
the handcrafted feature functions used in this work
by neural feature extractors trained jointly with
the policy. This would avoid the need to design
the feature functions and could be potentially extended to other modules of the SDS, making textto-action learning tractable. In addition, a single model can be potentially used in different domains (Papangelis and Stylianou, 2017), and different feudal architectures could make larger action spaces tractable (e.g. adding a third subpolicy to deal with actions dependent on 2 slots).

Acknowledgments
This research was funded by the EPSRC grant
EP/M018946/1 Open Domain Statistical Spoken
Dialogue Systems

References
Paweł Budzianowski, Stefan Ultes, Pei-Hao Su, Nikola
Mrkšić, Tsung-Hsien Wen, Inigo Casanueva, Lina
M. Rojas Barahona, and Milica Gašić. 2017. Subdomain modelling for dialogue management with hierarchical reinforcement learning. In Proc of SIGDIAL.
Iñigo Casanueva, Paweł Budzianowski, Pei-Hao Su,
Nikola Mrkšić, Tsung-Hsien Wen, Stefan Ultes,
Lina Rojas-Barahona, Steve Young, and Milica
Gašić. 2017. A benchmarking environment for reinforcement learning based task oriented dialogue
management. arXiv preprint arXiv:1711.11023 .
Inigo Casanueva, Thomas Hain, Heidi Christensen, Ricard Marxer, and Phil Green. 2015. Knowledge
transfer between speakers for personalised dialogue
management. In Proceedings of the 16th Annual
Meeting of the Special Interest Group on Discourse
and Dialogue. pages 12–21.
Heriberto Cuayáhuitl, Steve Renals, Oliver Lemon, and
Hiroshi Shimodaira. 2010. Evaluation of a hierar-

chical reinforcement learning spoken dialogue system. Computer Speech & Language 24(2):395–429.
Heriberto Cuayáhuitl, Seunghak Yu, Ashley
Williamson, and Jacob Carse. 2016. Deep reinforcement learning for multi-domain dialogue
systems. NIPS Workkshop .
Peter Dayan and Geoffrey E Hinton. 1993. Feudal reinforcement learning. In Advances in neural information processing systems. pages 271–278.
Milica Gašić, Catherine Breslin, Matthew Henderson,
Dongho Kim, Martin Szummer, Blaise Thomson,
Pirros Tsiakoulis, and Steve Young. 2013. Pomdpbased dialogue manager adaptation to extended domains. In Proceedings of the SIGDIAL Conference.
Milica Gašić, Nikola Mrkšić, Pei-hao Su, David
Vandyke, Tsung-Hsien Wen, and Steve Young.
2015. Policy committee for adaptation in multidomain spoken dialogue systems. In Automatic
Speech Recognition and Understanding (ASRU),
2015 IEEE Workshop on. IEEE, pages 806–812.
James Henderson, Oliver Lemon, and Kallirroi
Georgila. 2008. Hybrid reinforcement/supervised
learning of dialogue policies from fixed data sets.
Computational Linguistics 34(4):487–511.
M. Henderson, B. Thomson, and J. Williams. 2014a.
The Second Dialog State Tracking Challenge. In
Proc of SIGdial.
M. Henderson, B. Thomson, and S. J. Young. 2014b.
Word-based Dialog State Tracking with Recurrent
Neural Networks. In Proc of SIGdial.
Esther Levin, Roberto Pieraccini, and Wieland Eckert.
1998. Using markov decision process for learning
dialogue strategies. In Acoustics, Speech and Signal
Processing, 1998. Proceedings of the 1998 IEEE International Conference on. IEEE, volume 1, pages
201–204.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari
with deep reinforcement learning. arXiv preprint
arXiv:1312.5602 .
Alexandros Papangelis and Yannis Stylianou. 2017.
Single-model multi-domain dialogue management
with deep learning. In International Workshop for
Spoken Dialogue Systems.
B. Peng, X. Li, L. Li, J. Gao, A. Celikyilmaz, S. Lee,
and K.-F. Wong. 2017. Composite Task-Completion
Dialogue System via Hierarchical Deep Reinforcement Learning. ArXiv e-prints .
Olivier Pietquin, Matthieu Geist, Senthilkumar Chandramohan, and Hervé Frezza-Buet. 2011. Sampleefficient batch reinforcement learning for dialogue
management optimization. ACM Transactions on
Speech and Language Processing (TSLP) 7(3):7.

Pei-Hao Su, Pawel Budzianowski, Stefan Ultes, Milica Gasic, and Steve Young. 2017. Sample-efficient
actor-critic reinforcement learning with supervised
data for dialogue management. Proceedings of SigDial .
Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina RojasBarahona, Stefan Ultes, David Vandyke, TsungHsien Wen, and Steve Young. 2016. Continuously learning neural dialogue management. arXiv
preprint arXiv:1606.02689 .
Richard S. Sutton and Andrew G. Barto. 1999. Reinforcement Learning: An Introduction. MIT Press.
Stefan Ultes, Lina M. Rojas-Barahona, Pei-Hao Su,
David Vandyke, Dongho Kim, Iñigo Casanueva,
Paweł Budzianowski, Nikola Mrkšić, Tsung-Hsien
Wen, Milica Gašić, and Steve J. Young. 2017. Pydial: A multi-domain statistical dialogue system
toolkit. In ACL Demo. Association of Computational Linguistics.

B

DIP features

This section gives a detailed description of the DIP
feature functions φdip (b, s) = ψ0 (b) ⊕ ψj (b) ⊕
ψd (b, s) used in this work. The differences with
the features used in (Wang et al., 2015) and (Papangelis and Stylianou, 2017) are the following:
• No priority or importance features are used.
• No Potential contribution to DB search features are used.
• The joint belief features ψj (b) are extended
to account for large-domain aspects.
Feature
function

Feature
description

Feature
size

ψ0 (b)

last user dialogue act (bin) *
DB search method (bin) *
# of requested slots (bin)
offer happened *
last action was Inform no venue *
normalised # of slots (1/# of slots)
normalised avg. slot length (1/avg. # of values)

7
6
5
1
1
1
1

ψj (b)

prob. of the top 3 values of bj
prob. of *NONE* value of bj
entropy of bj
diff. between top and 2nd value probs. (bin)
# of slots with top value not *NONE* (bin)

3
1
1
5
5

ψd (b, s)

prob. of the top 3 values of s
prob. of *NONE* value of s
diff. between top and 2nd value probs. (bin)
entropy of s
# of values of s with prob. > 0 (bin)
normalised slot length (1/# of values)
slot length (bin)
entropy of the distr. of values of s in the DB

3
1
5
1
5
1
10
1

Zhuoran Wang, Tsung-Hsien Wen, Pei-Hao Su,
and Yannis Stylianou. 2015. Learning domainindependent dialogue policies via ontology parameterisation. In SIGDIAL Conference. pages 412–416.
Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the
IEEE 101(5):1160–1179.

A

Feudal Dialogue Policy algorithm

Algorithm 1 Feudal Dialogue Policy
1: for each dialogue turn do
2:
observe b
3:
bm = φm (b)
4:
am = argmax Qm (bm , am )

total

am ∈Am

5:
6:
7:

if am == am
i then
bi = φi (b)
a = argmax Qi (bi , ai )

. drop to πi

ai ∈Ai

8:
9:
10:

else am == am
. drop to πd
d then
bs = φs (b) ∀s ∈ S
slot, act = argmax Qs (bs , ad )
s∈S,ad ∈Ad

a = join(slot, act)
end if
13:
execute a
14: end for
11:
12:

64

Table 3: List of features composing the DIP features.
the tag (bin) denotes that a binary encoding is used for
this feature. Some of the joint features ψj (b) are extracted from the joint belief bj , computed as the Cartesian product of the beliefs of the individual slots. *
denotes that these features exist in the original belief
state b.

