title: What makes a multi armed bandit problem contextual and how do frameworks like TensorFlow and Stable Baseline implement these
body: <p>I aim to build an agent in the multi armed bandit setting. As far as I understand it is contextual, because I have a state machine which the agent uses and knows of. Each state is a one armed bandit and has a certain reward probability which he doesnt know of the beginning.</p>
  <p>So I was wondering, while doing the tutorials in StableBaselines 3 and TensorFlow on agents, how does the contextual part play into these agents in the MAB setting. In the <a href="https://www.tensorflow.org/agents/tutorials/per_arm_bandits_tutorial#multi-armed_bandits_with_arm_features" rel="nofollow noreferrer">tf documentation</a> was a sentence which was kind of explaining:</p>
  <blockquote>
  <p>In the &quot;classic&quot; Contextual Multi-Armed Bandits setting, an agent receives a context vector (aka observation) at every time step and has to choose from a finite set of numbered actions (arms) so as to maximize its cumulative reward.</p>
  </blockquote>
  <p>So in my case it means the agent, which is &quot;standing&quot; infront of a bandit machine (being in a certain state x), can only reach a certain amount of other machine (traverse in the state machine to possible n connected states). And not like in the classic MAB problem where the agent can go to all bandits (states) at any time. So the agent uses the observation function to get a context vector with the information what possible actions he has. This what makes a bandit problem contextual am I right?</p>
  <p>In these two frameworks there are basically the three parts: agent, policy and environment. The environment would contain my statemachine. But how does the context vector part fit into the design? I would have to add it to the policy somehow. But afaik the policies are kind of finished implementations. I would have the change the whole algorithm within the policy? Or are there &quot;contextual policies&quot; which take these contextual settings into account? I havent found any deeper information on the StableBaselines 3 or TensorFlow documentations.</p>