Entropy 2025, 27, 62 21 of 33
of its current position. This was implemented by letting the probability of transitioning to a
location be 1 for the location corresponding to the action chosen:
 
# Set transition probabilities to 1 for the chosen location
# For action " Move to Center Location "
B[1][:,:,1] = [ 1 .0 1 .0 1 .0 1 .0
0 .0 0 .0 0 .0 0 .0
0 .0 0 .0 0 .0 0 .0
0 .0 0 .0 0 .0 0 .0 ]
# For action " Move to Right Arm "
B[1][:,:,2] = [ 0 .0 0 .0 0 .0 0 .0
1 .0 1 .0 1 .0 1 .0
0 .0 0 .0 0 .0 0 .0
0 .0 0 .0 0 .0 0 .0 ]
# For action " Move to Left Arm "
B[1][:,:,3] = [ 0 .0 0 .0 0 .0 0 .0
0 .0 0 .0 0 .0 0 .0
1 .0 1 .0 1 .0 1 .0
0 .0 0 .0 0 .0 0 .0 ]
# For action " Move to Cue Location "
B[1][:,:,4] = [ 0 .0 0 .0 0 .0 0 .0
0 .0 0 .0 0 .0 0 .0
0 .0 0 .0 0 .0 0 .0
1 .0 1 .0 1 .0 1 .0 ]
 
The reward condition factor in B here was rather simple, as we let the agent correctly
assume that the reward condition never changed (although one could have enabled B
learning to let the agent learn the transition probabilities of the environment). There were
two possible states for these state factors, so we implemented this as a 2× 2 identity matrix:
 
# Identity matrix representing agent belief that reward condition does not change
B[2][:,:,1] = [ 1 .0 0 .0
0 .0 1 .0 ]
 
Then, we built C, the prior preference over observations. C encoded preferences for each
observation in each modality: the location, reward and cue. The relative value of the entry
corresponding to a specific observation defined the preference; here, we set all values with
neutral preferences as 0, and used positive values for preferences and negative values for
aversions. The only observations over which the agent had preferences were in the reward
modality, where the rewards (index 2) were preferred and losses (index 3) were disliked.
The strength of the preference determined the explore–exploit balance of the agent; here,
we use 3 as the value:
 
# Preference over locations modality
C[1] = [0 .0 , 0 .0 , 0 .0 , 0 .0 ] #No preference
# Preference over reward modality
C[2] = [0 .0 , 3 .0 , -3 .0 ] # Rewards preferred , losses not preferred
# Preference over cue modality
C[3] = [0 .0 , 0 .0 ] #No preference
 
Then, we built D, which was the prior or initial belief over the environmental states. There
was a vector in D for each of the two state factors: the location state and the reward
condition. We let the agent start with a belief that it was at the centre location (index 1),
and gave uniform priors for the reward condition: