Entropy 2025, 27, 62 10 of 33
2.3. Action in Active Inference
As with perception, action in AIF is guided by the minimisation of free energy. How-
ever, instead of VFE being minimised directly, it is the free energy that is expected to occur
depending on the actions taken by the agent—the expected free energy or EFE—that is
minimised. As stated below, choosing actions that minimise the EFE leads to a natural
balance between exploration and exploitation, ensuring preferences are realised and am-
biguity about the environment is minimised. In AIF, policies π are sequences of actions
u. The policy length (also called the planning horizon or temporal depth) is the length of
the policies being considered. The total number of policies therefore depends on the policy
length and the number of different actions that can be made at each time step. An EFE is
assigned to each policy π (denoted as Gπ), where policies associated with a lower EFE are
then more likely to be chosen.
One can rewrite the EFE in different ways to highlight different consequences of
optimising it. Below, we show the two most crucial ways to rewrite it, taken from [1,33].
We denote the states and observations that are expected future outcomes of actions with
(~). Additionally, we introduce a preference priorC that encodes the agent’s preferences:
Gπ = −Eq(˜o,˜s|π)[ln q(˜s|˜o, π) − ln q(˜s|π)]
| {z }
Information gain
−Eq(˜o|π)[ln p(˜o|C)]
| {z }
Pragmatic value
(17)
The expression above shows how minimising the EFE leads to a natural balance between
information gathering and realising preferences. The first term on the right-hand side is the
change in belief from the prior to the posterior under a given policy called the epistemic
value or information gain. Optimising this value is what leads to (notably non-random)
exploratory behaviour. The second term is the pragmatic value; minimising this value
ensures that observations are in accordance with the preference prior C.
Another way to express the EFE is in terms of risk and ambiguity:
Gπ = Eq(˜s|π)[H(p(˜o|˜s))]
| {z }
Expected ambiguity
+ DKL [q(˜o|π)∥p(˜o|C)]| {z }
Risk (outcomes)
(18)
Here, the first term on the right-hand side captures the expected entropy, or uncertainty,
of the outcomes given the environmental states. Minimising this quantity ensures that
the agent will seek states where observations can most clearly be used to distinguish
between environmental states. The second term is the KL divergence of the expected
observations from preferred observations, capturing the risk of making unwanted (i.e., a
priori surprising) observations, which is also minimised by minimising the EFE.
2.4. Learning in Active Inference
In AIF, the parameters of the generative model can also be updated via Bayesian-belief-
updating methods, a process called “parameter learning” or sometimes just “learning” [2].
In general, this is performed by introducing belief distributions over the possible values
of the parameters that are subject to learning, and updating this distribution for each
observation using Bayesian belief updating. This additionally implies introducing priors
on the belief distributions. Depending on the type of generative model used, the belief
distributions and their priors will take different forms, and so will their update equations.
In the following, we demonstrate parameter learning specifically in the context of POMDPs.
The parameters that are subject to learning in POMDPs are usually the entries in the
five matrices. Since the matrices consist of categorical probability distributions, it is natural
to use Dirichlet distributions—distributions over categorical probability distributions—as
belief distributions over their values [ 33,52]. Beliefs about each probability distribution