Implementing Policy iteration methods in Open AI Gym
No, OpenAI Gym environments will not provide you with the information in that form. In order to collect that information you will need to explore the environment via sampling: i.e. selecting actions and receiving observations and rewards. With these samples you can estimate them.
One basic way to approximate these values is to use LSPI (least square policy iteration), as far as I remember, you will find more about this in Sutton too.
I am currently reading "Reinforcement Learning" from Sutton & Barto and I am attempting to write some of the methods myself.
Policy iteration is the one I am currently working on. I am trying to use OpenAI Gym for a simple problem, such as CartPole or continuous mountain car. 
However, for policy iteration, I need both the transition matrix between states and the Reward matrix. 
Are these available from the 'environment' that you build in OpenAI Gym. 
I am using python. 
If not, how do I calculate these values, and use the environment?