What is the difference between environment states and agent states in terms of Markov property?
$\mathcal S$ is just a set of all possible states. It doesn't matter if it's agents perceived state or true environment state, they are within the same set of states. Agent cannot perceive itself to be in some &quot;middle&quot; state that's not in $\mathcal S$, it might think that's in the state that's not the actual environment state but that state is also in set of all states.
To give an example, if the car can be blue or red, then the agent might think that the state is blue car or red car, but it cannot think that car is purple because that's not one of the possible states. It might wrongly think that the car is blue when the actual car is red, but that's ok because blue is one of the possible car states. Of course, it might also correctly think that the car state is red.