Entropy 2025, 27, 62 20 of 33
 
# Set center ( column 1) and cue ( column 4) to give no reward observation ( row 1)
# Set reward and loss probabilities to 0.5 for the arm locations
# For reward condition right
A[2][:,:,1] = [ 1 .0 0 .0 0 .0 1 .0
0 .0 0 .5 0 .5 0 .0
0 .0 0 .5 0 .5 0 .0 ]
# For reward condition left
A[2][:,:,2] = [ 1 .0 0 .0 0 .0 1 .0
0 .0 0 .5 0 .5 0 .0
0 .0 0 .5 0 .5 0 .0 ]
 
Figure 3. Reward probabilities for the four locations. The centre (column 1) and cue (column 4)
locations always resulted in the “no reward” observation (row 1). The two arms (columns 3 and
4) resulted in either rewards (row 2) or losses (row 3), with some probability. Left: the agent’s
agnostic starting beliefs about reward probabilities. Right: the true reward probabilities for the
reward condition left arm, which the agent needed to learn over time. The amount of saturation of
the green color represent the likelihood of a specific observation in a give state.
The third and last modality was the cue modality, which mapped the cue observations
onto the location and reward condition states. This resulted in an A that was two cue
observations by four location states by two reward conditions, i.e., a 2 × 4 × 2 tensor. The
observations in this modality were correctly assumed by the agent to truthfully reveal the
current reward condition—i.e., whether the right or left arm was better—when standing at
the cue location. We implemented this by giving each cue observation equal probabilities
at all locations except the cue location, where there was a perfect correspondence between
the reward condition and the observation:
 
# Set cue observation probabilities to be equal at all
# locations except the cue location ( column 4).
# Let cue locations correspond to reward conditions .
# For reward condition right
A[3][:,:,1] = [ 0 .5 0 .5 0 .5 1 .0
0 .5 0 .5 0 .5 0 .0 ]
# For reward condition left
A[3][:,:,2] = [ 0 .5 0 .5 0 .5 0 .0
0 .5 0 .5 0 .5 1 .0 ]
 
Having created all three modalities of A, we could continue to B, or the transition model.
Each of the two state factors of B—the location factor and reward condition factor—needed
to be defined separately. We started with the location factor, which contained the transition
to and from four possible location states under four different actions: a 4 × 4 × 4 tensor.
The agent could control these states perfectly with its four movement actions, independently