class_docstr: Interface for environments with sub-step structure, which is generally enough to cover multi-step,
  hierarchical and multi-agent environments.
  
  This environment can continuously create and destroy a previously unknown,
  unlimited number of actors during the course of an episode. Every actor is associated with one of the
  available policies.
  
  The lifecycle of the environment is decoupled from the lifecycle of the actors. The interaction loop should
  continue, until the environment as a whole is set to done, which is returned as usual by the step() function.
  Individual actors might end earlier, which can be queried by the is_actor_done() method.
  
  Pseudo-code of the interaction loop:
  
  # start a new episode
  observation = env.reset()
  
  while not done:
      # find out which actor is next to act (dictated by the env)
      sub_step_key, actor_id = env.actor_id()
  
      # obtain the next action from the policy
      action = sample_from_policy(observation, sub_step_key, actor_id)
  
      # step the env
      observation, reward, done, info = env.step(action)
  
      # optionally use is_actor_done() to find out if the actor was terminated (relevant during training)
class_docstr_tok: ["Interface","for","environments","with","sub-step","structure",",","which","is","generally","enough","to","cover","multi-step",",","hierarchical","and","multi-agent","environments",".","This","environment","can","continuously","create","and","destroy","a","previously","unknown",",","unlimited","number","of","actors","during","the","course","of","an","episode",".","Every","actor","is","associated","with","one","of","the","available","policies",".","The","lifecycle","of","the","environment","is","decoupled","from","the","lifecycle","of","the","actors",".","The","interaction","loop","should","continue",",","until","the","environment","as","a","whole","is","set","to","done",",","which","is","returned","as","usual","by","the","step","(",")","function",".","Individual","actors","might","end","earlier",",","which","can","be","queried","by","the","is_actor_done","(",")","method",".","Pseudo-code","of","the","interaction","loop",":","#","start","a","new","episode","observation","=","env.reset","(",")","while","not","done",":","#","find","out","which","actor","is","next","to","act","(","dictated","by","the","env",")","sub_step_key",",","actor_id","=","env.actor_id","(",")","#","obtain","the","next","action","from","the","policy","action","=","sample_from_policy","(","observation",",","sub_step_key",",","actor_id",")","#","step","the","env","observation",",","reward",",","done",",","info","=","env.step","(","action",")","#","optionally","use","is_actor_done","(",")","to","find","out","if","the","actor","was","terminated","(","relevant","during","training",")"]
docstr: Returns the maximum count of agents per sub-step that the environment features.
  
  If the agent count for a particular sub-step is dynamic (unknown upfront), then returns -1 for this sub-step.
  This then limits available training configurations (e.g. with a dynamic number of agents, only
  a shared policy can be trained -- not multiple separate per-agent policies, as their count is not
  known upfront).
  
  For example:
    - For a vehicle-routing environment where max 5 driver agents will get to act during sub-step 0,
      this method should return {0: 5}
    - For a vehicle routing environment where a dynamic number of agents will get to act during sub-step 0,
      this method should return {0: -1}
    - For a two-step cutting environment where a piece is selected during sub-step 0 and then cut during
      sub-step 1 (with just one selection and cut happening in each step),
      this method should return {0: 1, 1: 1}
docstr_tok: ["Returns","the","maximum","count","of","agents","per","sub-step","that","the","environment","features",".","If","the","agent","count","for","a","particular","sub-step","is","dynamic","(","unknown","upfront",")",",","then","returns","-1","for","this","sub-step",".","This","then","limits","available","training","configurations","(","e.g",".","with","a","dynamic","number","of","agents",",","only","a","shared","policy","can","be","trained","--","not","multiple","separate","per-agent","policies",",","as","their","count","is","not","known","upfront",")",".","For","example",":","-","For","a","vehicle-routing","environment","where","max","5","driver","agents","will","get","to","act","during","sub-step","0",",","this","method","should","return","{","0",":","5","}","-","For","a","vehicle","routing","environment","where","a","dynamic","number","of","agents","will","get","to","act","during","sub-step","0",",","this","method","should","return","{","0",":","-1","}","-","For","a","two-step","cutting","environment","where","a","piece","is","selected","during","sub-step","0","and","then","cut","during","sub-step","1","(","with","just","one","selection","and","cut","happening","in","each","step",")",",","this","method","should","return","{","0",":","1",",","1",":","1","}"]
code: def agent_counts_dict(self) -> Dict[StepKeyType, int]:
      """Returns the maximum count of agents per sub-step that the environment features.
  
      If the agent count for a particular sub-step is dynamic (unknown upfront), then returns -1 for this sub-step.
      This then limits available training configurations (e.g. with a dynamic number of agents, only
      a shared policy can be trained -- not multiple separate per-agent policies, as their count is not
      known upfront).
  
      For example:
        - For a vehicle-routing environment where max 5 driver agents will get to act during sub-step 0,
          this method should return {0: 5}
        - For a vehicle routing environment where a dynamic number of agents will get to act during sub-step 0,
          this method should return {0: -1}
        - For a two-step cutting environment where a piece is selected during sub-step 0 and then cut during
          sub-step 1 (with just one selection and cut happening in each step),
          this method should return {0: 1, 1: 1}
      """
  
file_path: maze/maze/core/env/structured_env.py
imports: from abc import abstractmethod
  from typing import Union, Dict, NamedTuple, Optional
  import numpy
  from maze.core.env.base_env import BaseEnv
label_desc: Use image node_id 3 for calling the StructuredEnv obj's underlying member method code with example usage: obj.agent_counts_dict() without return types