class_docstr: Maps policy IDs to Policy objects.
  
  Thereby, keeps n policies in memory and - when capacity is reached -
  writes the least recently used to disk. This allows adding 100s of
  policies to a Algorithm for league-based setups w/o running out of memory.
docstr: Initializes a PolicyMap instance.
  
  Args:
      capacity: The size of the Policy object cache. This is the maximum number
          of policies that are held in RAM memory. When reaching this capacity,
          the least recently used Policy's state will be stored in the Ray object
          store and recovered from there when being accessed again.
      policy_states_are_swappable: Whether all Policy objects in this map can be
          "swapped out" via a simple `state = A.get_state(); B.set_state(state)`,
          where `A` and `B` are policy instances in this map. You should set
          this to True for significantly speeding up the PolicyMap's cache lookup
          times, iff your policies all share the same neural network
          architecture and optimizer types. If True, the PolicyMap will not
          have to garbage collect old, least recently used policies, but instead
          keep them in memory and simply override their state with the state of
          the most recently accessed one.
          For example, in a league-based training setup, you might have 100s of
          the same policies in your map (playing against each other in various
          combinations), but all of them share the same state structure
          (are "swappable").
docstr_tok: ["Initializes","a","PolicyMap","instance",".","Args",":","capacity",":","The","size","of","the","Policy","object","cache",".","This","is","the","maximum","number","of","policies","that","are","held","in","RAM","memory",".","When","reaching","this","capacity",",","the","least","recently","used","Policy","'s","state","will","be","stored","in","the","Ray","object","store","and","recovered","from","there","when","being","accessed","again",".","policy_states_are_swappable",":","Whether","all","Policy","objects","in","this","map","can","be","``","swapped","out","''","via","a","simple","`","state","=","A.get_state","(",")",";","B.set_state","(","state",")","`",",","where","`","A","`","and","`","B","`","are","policy","instances","in","this","map",".","You","should","set","this","to","True","for","significantly","speeding","up","the","PolicyMap","'s","cache","lookup","times",",","iff","your","policies","all","share","the","same","neural","network","architecture","and","optimizer","types",".","If","True",",","the","PolicyMap","will","not","have","to","garbage","collect","old",",","least","recently","used","policies",",","but","instead","keep","them","in","memory","and","simply","override","their","state","with","the","state","of","the","most","recently","accessed","one",".","For","example",",","in","a","league-based","training","setup",",","you","might","have","100s","of","the","same","policies","in","your","map","(","playing","against","each","other","in","various","combinations",")",",","but","all","of","them","share","the","same","state","structure","(","are","``","swappable","''",")","."]
code: def __init__(
      self,
      *,
      capacity: int = 100,
      policy_states_are_swappable: bool = False,
      # Deprecated args.
      worker_index=None,
      num_workers=None,
      policy_config=None,
      session_creator=None,
      seed=None,
  ):
      """Initializes a PolicyMap instance.
  
      Args:
          capacity: The size of the Policy object cache. This is the maximum number
              of policies that are held in RAM memory. When reaching this capacity,
              the least recently used Policy's state will be stored in the Ray object
              store and recovered from there when being accessed again.
          policy_states_are_swappable: Whether all Policy objects in this map can be
              "swapped out" via a simple `state = A.get_state(); B.set_state(state)`,
              where `A` and `B` are policy instances in this map. You should set
              this to True for significantly speeding up the PolicyMap's cache lookup
              times, iff your policies all share the same neural network
              architecture and optimizer types. If True, the PolicyMap will not
              have to garbage collect old, least recently used policies, but instead
              keep them in memory and simply override their state with the state of
              the most recently accessed one.
              For example, in a league-based training setup, you might have 100s of
              the same policies in your map (playing against each other in various
              combinations), but all of them share the same state structure
              (are "swappable").
      """
      if policy_config is not None:
          deprecation_warning(
              old="PolicyMap(policy_config=..)",
              error=True,
          )
  
      super().__init__()
  
      self.capacity = capacity
  
      if any(
          i is not None
          for i in [
              policy_config,
              worker_index,
              num_workers,
              session_creator,
              seed,
          ]
      ):
          deprecation_warning(
              old="PolicyMap([deprecated args]...)",
              new="PolicyMap(capacity=..., policy_states_are_swappable=...)",
              error=False,
          )
  
      self.policy_states_are_swappable = policy_states_are_swappable
  
      # The actual cache with the in-memory policy objects.
      self.cache: Dict[str, Policy] = {}
  
      # Set of keys that may be looked up (cached or not).
      self._valid_keys: Set[str] = set()
      # The doubly-linked list holding the currently in-memory objects.
      self._deque = deque()
  
      # Ray object store references to the stashed Policy states.
      self._policy_state_refs = {}
  
      # Lock used for locking some methods on the object-level.
      # This prevents possible race conditions when accessing the map
      # and the underlying structures, like self._deque and others.
      self._lock = threading.RLock()
  
imports: from collections import deque
  import threading
  from typing import Dict, Set
  import logging
  import ray
  from ray.rllib.policy.policy import Policy
  from ray.rllib.utils.annotations import override
  from ray.rllib.utils.deprecation import deprecation_warning
  from ray.rllib.utils.framework import try_import_tf
  from ray.rllib.utils.threading import with_lock
  from ray.rllib.utils.typing import PolicyID
  from ray.util.annotations import PublicAPI
label_desc: Use image node_id 1 to create a new PolicyMap object from inherited base classes: dict with example: obj = PolicyMap()