Entropy 2025, 27, 62 5 of 33
quantities as its target: the variational free energy (VFE) in the case of perception and the
expected free energy (EFE) in the case of action. The VFE is the free energy associated with
a given sensory observation and is resolved perceptually by updating beliefs about the
environment. The EFE is the free energy that is expected in the future, contingent on a
given policy or course of action. Choosing action policies associated with a low EFE lead
to reducing uncertainty about the environment, as well as making preferred observations
more likely.
2.1. POMDPs in Active Inference
In AIF, the POMDP is one of the most common families of generative models used
to make inferences about the environment. It is a Markovian discrete state-space model,
where employing it means representing the environment and observations as inhabiting
one among a set of possible (possibly multidimensional) states, and that the changes
in these states can only depend on the system’s previous state and the agent’s actions.
Environmental states are not directly observable, so they have to be inferred based on
incoming sensory observations. In AIF for POMDPs and other generative models in general,
both perception and action are cast as Bayesian inferences (see Sections 2.2 and 2.3), as well
as the learning of parameters of the generative model (see Section 2.4). Crucially, an agent’s
generative model does not a priori have to be isomorphic to the true environment (i.e.,
the data-generating process), although this will generally lead to a successful inference,
and that the generative model will therefore often come to resemble the environment
through learning.
A discrete state-space POMDP in AIF is conventionally defined by five main sets of
parameters: A, B, C, D and E [1,33], see Figure 1. Together, these parametrise the agent’s
prior beliefs about the prior probability of different states in the environment, how states
of the environment change and how they generate observations. Typically, they will be
vectors, matrices or tensors; however, henceforth we denote them by their corresponding
letter in bold. These make up the components needed for the agent to perform AIF.
A, also called the observation model, represents the state-to-observation likelihood model.
This describes how observations depend on or are generated by states of the environment.
It is structured as a matrix with a column for each possible environmental states, and a row
for each possible observation o. Each column is then a categorical probability distribution
over the observations that will occur given the environmental state (meaning that each
column must contain non-negative values that sum to 1). If the observations are multidi-
mensional (i.e., multiple observations are made at each time point), there is a matrix for
each observation modality. If two or more states determine the observation, the likelihood
model then becomes a tensor. IfA is imprecise (i.e., the probabilities are highly entropic and
evenly distributed), observations are taken to carry less information about the environment,
in many cases leading to more uncertain inferences, and vice versa.
B, also called the transition model, describes the state-to-state transition probabilities of
environmental states s. B encodes the agent’s assumptions about how the environment
changes over time, depending on its actions. It has a column and a row for each environ-
mental state s, where each column is a categorical probability distribution over the states
the environment will take on the next time step, given the state it is currently in. If the envi-
ronment is modelled as multidimensional, there will be a matrix for each environmental
state factor. Additionally, there is a separate matrix for each possible action (making each
factor in B a tensor). This means that for every factor in the model, there may be one or