docstr: Basic implementation of group job resources calculation
  
  Each toposort level is individually sorted into a series of layers, where each
  layer fits within the constraints. Resource constraints represent a "width" into
  which the layer must fit. For instance, with a mem_mb constraint of 5G, all the
  jobs in a single layer must together not consume more than 5G of memory. Any
  jobs that would exceed this constraint are pushed into a new layer. The overall
  width for the entire group job is equal to the width of the widest layer.
  
  Additive resources (by default, "runtime") represent the "height" of the layer.
  They are not directly constrained, but their value will be determined by the
  sorting of jobs based on other constraints. Each layer's height is equal to the
  height of its tallest job. For instance, a layer containing a 3hr job will have
  a runtime height of 3 hr. The total height of the entire group job will be the
  sum of the heights of all the layers.
  
  Note that both height and width are multidimensial, so layer widths will be
  calculated with respect to every constraint created by the user.
  
  In this implementation, there is no mixing of layers, which may lead to "voids".
  For instance, a layer containing a tall, skinny job of 3hr length and 1G mem
  combined with a short, fat job of 10min length and 20G memory would have a 2hr
  50min period where 19G of memory are not used. In practice, this void will be
  filled by the actual snakemake subprocess, which performs real-time scheduling
  of jobs as resources become available.  But it may lead to overestimation of
  resource requirements.
  
  To help mitigate against voids, this implementation sorts the jobs within a
  toposort level before assignment to layers. Jobs are first sorted by their
  overall width relative to the available constraints. So the fattest jobs will
  grouped together on the same layer. Jobs are then sorted by the resources
  specified in ``sortby``, by default "runtime". So jobs of similar length will be
  grouped on the same layer.
  
  Users can help mitigate against voids by grouping jobs of similar resource
  dimensions.  Eclectic groups of various runtimes and resource consumptions will
  not be estimated as efficiently as groups of homogenous consumptions.
  
  Parameters
  ----------
  toposorted_jobs : list of lists of jobs
      Jobs sorted into toposort levels: the jobs in each level only depend on jobs
      in previous levels.
  constraints : dict of str -> int
      Upper limit of resource allowed. Resources without constraints will be
      treated as infinite
  run_local : bool
      True if the group is being run in the local process, rather than being
      submitted. Relevant for Pipe groups and Service groups
  additive_resources : list of str, optional
      Resources to be treated as the "height" of each layer, i.e. to be summed
      across layers.
  sortby : list of str, optional
      Resources by which to sort jobs prior to layer assignment.
  
  Returns
  -------
  Dict of str -> int,str
      Total resource requirements of the group job
  
  Raises
  ------
  WorkflowError
      Raised if an individual job requires more resources than the constraints
      allow (chiefly relevant for pipe groups)
docstr_tok: ["Basic","implementation","of","group","job","resources","calculation","Each","toposort","level","is","individually","sorted","into","a","series","of","layers",",","where","each","layer","fits","within","the","constraints",".","Resource","constraints","represent","a","``","width","''","into","which","the","layer","must","fit",".","For","instance",",","with","a","mem_mb","constraint","of","5G",",","all","the","jobs","in","a","single","layer","must","together","not","consume","more","than","5G","of","memory",".","Any","jobs","that","would","exceed","this","constraint","are","pushed","into","a","new","layer",".","The","overall","width","for","the","entire","group","job","is","equal","to","the","width","of","the","widest","layer",".","Additive","resources","(","by","default",",","``","runtime","''",")","represent","the","``","height","''","of","the","layer",".","They","are","not","directly","constrained",",","but","their","value","will","be","determined","by","the","sorting","of","jobs","based","on","other","constraints",".","Each","layer","'s","height","is","equal","to","the","height","of","its","tallest","job",".","For","instance",",","a","layer","containing","a","3hr","job","will","have","a","runtime","height","of","3","hr",".","The","total","height","of","the","entire","group","job","will","be","the","sum","of","the","heights","of","all","the","layers",".","Note","that","both","height","and","width","are","multidimensial",",","so","layer","widths","will","be","calculated","with","respect","to","every","constraint","created","by","the","user",".","In","this","implementation",",","there","is","no","mixing","of","layers",",","which","may","lead","to","``","voids","''",".","For","instance",",","a","layer","containing","a","tall",",","skinny","job","of","3hr","length","and","1G","mem","combined","with","a","short",",","fat","job","of","10min","length","and","20G","memory","would","have","a","2hr","50min","period","where","19G","of","memory","are","not","used",".","In","practice",",","this","void","will","be","filled","by","the","actual","snakemake","subprocess",",","which","performs","real-time","scheduling","of","jobs","as","resources","become","available",".","But","it","may","lead","to","overestimation","of","resource","requirements",".","To","help","mitigate","against","voids",",","this","implementation","sorts","the","jobs","within","a","toposort","level","before","assignment","to","layers",".","Jobs","are","first","sorted","by","their","overall","width","relative","to","the","available","constraints",".","So","the","fattest","jobs","will","grouped","together","on","the","same","layer",".","Jobs","are","then","sorted","by","the","resources","specified","in","``","sortby","``",",","by","default","``","runtime","''",".","So","jobs","of","similar","length","will","be","grouped","on","the","same","layer",".","Users","can","help","mitigate","against","voids","by","grouping","jobs","of","similar","resource","dimensions",".","Eclectic","groups","of","various","runtimes","and","resource","consumptions","will","not","be","estimated","as","efficiently","as","groups","of","homogenous","consumptions",".","Parameters","--","--","--","--","--","toposorted_jobs",":","list","of","lists","of","jobs","Jobs","sorted","into","toposort","levels",":","the","jobs","in","each","level","only","depend","on","jobs","in","previous","levels",".","constraints",":","dict","of","str","-",">","int","Upper","limit","of","resource","allowed",".","Resources","without","constraints","will","be","treated","as","infinite","run_local",":","bool","True","if","the","group","is","being","run","in","the","local","process",",","rather","than","being","submitted",".","Relevant","for","Pipe","groups","and","Service","groups","additive_resources",":","list","of","str",",","optional","Resources","to","be","treated","as","the","``","height","''","of","each","layer",",","i.e",".","to","be","summed","across","layers",".","sortby",":","list","of","str",",","optional","Resources","by","which","to","sort","jobs","prior","to","layer","assignment",".","Returns","--","--","--","-","Dict","of","str","-",">","int",",","str","Total","resource","requirements","of","the","group","job","Raises","--","--","--","WorkflowError","Raised","if","an","individual","job","requires","more","resources","than","the","constraints","allow","(","chiefly","relevant","for","pipe","groups",")"]
code: def basic_layered(
      cls,
      toposorted_jobs,
      constraints,
      run_local,
      additive_resources=None,
      sortby=None,
  ):
      """Basic implementation of group job resources calculation
  
      Each toposort level is individually sorted into a series of layers, where each
      layer fits within the constraints. Resource constraints represent a "width" into
      which the layer must fit. For instance, with a mem_mb constraint of 5G, all the
      jobs in a single layer must together not consume more than 5G of memory. Any
      jobs that would exceed this constraint are pushed into a new layer. The overall
      width for the entire group job is equal to the width of the widest layer.
  
      Additive resources (by default, "runtime") represent the "height" of the layer.
      They are not directly constrained, but their value will be determined by the
      sorting of jobs based on other constraints. Each layer's height is equal to the
      height of its tallest job. For instance, a layer containing a 3hr job will have
      a runtime height of 3 hr. The total height of the entire group job will be the
      sum of the heights of all the layers.
  
      Note that both height and width are multidimensial, so layer widths will be
      calculated with respect to every constraint created by the user.
  
      In this implementation, there is no mixing of layers, which may lead to "voids".
      For instance, a layer containing a tall, skinny job of 3hr length and 1G mem
      combined with a short, fat job of 10min length and 20G memory would have a 2hr
      50min period where 19G of memory are not used. In practice, this void will be
      filled by the actual snakemake subprocess, which performs real-time scheduling
      of jobs as resources become available.  But it may lead to overestimation of
      resource requirements.
  
      To help mitigate against voids, this implementation sorts the jobs within a
      toposort level before assignment to layers. Jobs are first sorted by their
      overall width relative to the available constraints. So the fattest jobs will
      grouped together on the same layer. Jobs are then sorted by the resources
      specified in ``sortby``, by default "runtime". So jobs of similar length will be
      grouped on the same layer.
  
      Users can help mitigate against voids by grouping jobs of similar resource
      dimensions.  Eclectic groups of various runtimes and resource consumptions will
      not be estimated as efficiently as groups of homogenous consumptions.
  
      Parameters
      ----------
      toposorted_jobs : list of lists of jobs
          Jobs sorted into toposort levels: the jobs in each level only depend on jobs
          in previous levels.
      constraints : dict of str -> int
          Upper limit of resource allowed. Resources without constraints will be
          treated as infinite
      run_local : bool
          True if the group is being run in the local process, rather than being
          submitted. Relevant for Pipe groups and Service groups
      additive_resources : list of str, optional
          Resources to be treated as the "height" of each layer, i.e. to be summed
          across layers.
      sortby : list of str, optional
          Resources by which to sort jobs prior to layer assignment.
  
      Returns
      -------
      Dict of str -> int,str
          Total resource requirements of the group job
  
      Raises
      ------
      WorkflowError
          Raised if an individual job requires more resources than the constraints
          allow (chiefly relevant for pipe groups)
      """
      additive_resources = (
          additive_resources
          if additive_resources is not None
          else ["runtime"]
      )
      sortby = sortby if sortby is not None else ["runtime"]
  
      total_resources = defaultdict(int)
      total_resources["_nodes"] = 1
      blocks = []
      # iterate over siblings that can be executed in parallel
      for siblings in toposorted_jobs:
          # Total resource requirements for this toposort layer
          block_resources = {}
  
          job_resources = []
          pipe_resources = defaultdict(list)
          for job in siblings:
              # Get resources, filtering out FileNotFoundErrors. List items will
              # be job resources objects with (resource: value)
              # [
              #   { "runtime": 5, "threads": 2, "tmpdir": "/tmp" },
              #   { "runtime": 15, "tmpdir": "/tmp"},
              #   ...
              # ]
              # Pipe jobs and regular jobs are put in seperate lists.
              try:
                  # Remove any TBDStrings from values. These will typically arise
                  # here because the default mem_mb and disk_mb are based off of
                  # input file size, and intermediate files in the group are not yet
                  # generated. Thus rules consuming such files must explicitely
                  # specify their resources
                  res = {
                      k: res
                      for k, res, in job.resources.items()
                      if not isinstance(res, TBDString)
                  }
                  if job.pipe_group:
                      pipe_resources[job.pipe_group].append(res)
                  else:
                      job_resources.append(res)
              except FileNotFoundError:
                  # Skip job if resource evaluation leads to a file not found error.
                  # This will be caused by an inner job, which needs files created by
                  # the same group. All we can do is to ignore such jobs for now.
                  continue
  
          # Jobs in pipe groups must be run simultaneously, so we merge all the
          # resources of each pipe group into one big "job". Resources are combined
          # as "intralayer", so additives (like runtime) are maxed, and the rest are
          # summed
          for pipe in pipe_resources.values():
              job_resources.append(
                  cls._merge_resource_dict(
                      pipe,
                      methods={res: max for res in additive_resources},
                      default_method=sum,
                  )
              )
  
          # Set of resource types requested in at least one job
          resource_types = list(set(it.chain(*job_resources)))
          int_resources = {}
          # Sort all integer resources in job_resources into int_resources. Resources
          # defined as a string are placed immediately into block_resources.
          for res in resource_types:
              if res == "_nodes":
                  continue
  
              values = [
                  resources.get(res, 0) for resources in job_resources
              ]
  
              if cls._is_string_resource(res, values):
                  block_resources[res] = values[0]
              else:
                  int_resources[res] = values
  
          # Collect values from global_resources to use as constraints.
          sorted_constraints = {
              name: constraints.get(name, None)
              for name in int_resources
          }
  
          # For now, we are unable to handle a constraint on runtime, so ignore.
          # Jobs requesting too much runtime will still get flagged by the
          # scheduler
          for res in additive_resources:
              if res in sorted_constraints:
                  sorted_constraints[res] = None
  
          # Get layers
          try:
              layers = cls._get_layers(
                  int_resources, sorted_constraints.values(), sortby
              )
          except WorkflowError as err:
              raise cls._get_saturated_resource_error(
                  additive_resources, err.args[0]
              )
  
          # Merge jobs within layers
          intralayer_merge_methods = [
              max if res in additive_resources else sum
              for res in int_resources
          ]
          merged = [
              cls._merge_resource_layer(layer, intralayer_merge_methods)
              for layer in layers
          ]
  
          # Combine layers
          interlayer_merge_methods = [
              sum if res in additive_resources else max
              for res in int_resources
          ]
          combined = cls._merge_resource_layer(
              merged, interlayer_merge_methods
          )
  
          # Reassign the combined values from each layer to their resource names
          block_resources.update(dict(zip(int_resources, combined)))
  
          blocks.append(block_resources)
  
      if run_local:
          return {
              **cls._merge_resource_dict(blocks, default_method=sum),
              "_nodes": 1,
          }
  
      return {
          **cls._merge_resource_dict(
              blocks,
              default_method=max,
              methods={res: sum for res in additive_resources},
          ),
          "_nodes": 1,
      }
  
imports: from collections import UserDict, defaultdict
  import itertools
  import operator
  import re
  import tempfile
  from snakemake.exceptions import ResourceScopesException, WorkflowError
  from snakemake.common.tbdstring import TBDString
label_desc: Use image node_id 1 for calling the GroupResources obj's underlying member method code with example usage: obj.basic_layered(cls, toposorted_jobs, constraints, run_local, additive_resources, sortby) and returns: dict, dict