class_docstr: Slurm job executor
  This class is used to hold the parameters to run a job on slurm.
  In practice, it will create a batch file in the specified directory for each job,
  and pickle the task function and parameters. At completion, the job will also pickle
  the output. Logs are also dumped in the same directory.
  
  Parameters
  ----------
  folder: Path/str
      folder for storing job submission/output and logs.
  max_num_timeout: int
      Maximum number of time the job can be requeued after timeout (if
      the instance is derived from helpers.Checkpointable)
  python: Optional[str]
      Command to launch python. This allow to use singularity for example.
      Caller is responsible to provide a valid shell command here.
      By default reuse the current python executable
  
  Note
  ----
  - be aware that the log/output folder will be full of logs and pickled objects very fast,
    it may need cleaning.
  - the folder needs to point to a directory shared through the cluster. This is typically
    not the case for your tmp! If you try to use it, slurm will fail silently (since it
    will not even be able to log stderr.
  - use update_parameters to specify custom parameters (n_gpus etc...). If you
    input erroneous parameters, an error will print all parameters available for you.
class_docstr_tok: ["Slurm","job","executor","This","class","is","used","to","hold","the","parameters","to","run","a","job","on","slurm",".","In","practice",",","it","will","create","a","batch","file","in","the","specified","directory","for","each","job",",","and","pickle","the","task","function","and","parameters",".","At","completion",",","the","job","will","also","pickle","the","output",".","Logs","are","also","dumped","in","the","same","directory",".","Parameters","--","--","--","--","--","folder",":","Path\/str","folder","for","storing","job","submission\/output","and","logs",".","max_num_timeout",":","int","Maximum","number","of","time","the","job","can","be","requeued","after","timeout","(","if","the","instance","is","derived","from","helpers.Checkpointable",")","python",":","Optional","[","str","]","Command","to","launch","python",".","This","allow","to","use","singularity","for","example",".","Caller","is","responsible","to","provide","a","valid","shell","command","here",".","By","default","reuse","the","current","python","executable","Note","--","--","-","be","aware","that","the","log\/output","folder","will","be","full","of","logs","and","pickled","objects","very","fast",",","it","may","need","cleaning",".","-","the","folder","needs","to","point","to","a","directory","shared","through","the","cluster",".","This","is","typically","not","the","case","for","your","tmp","!","If","you","try","to","use","it",",","slurm","will","fail","silently","(","since","it","will","not","even","be","able","to","log","stderr",".","-","use","update_parameters","to","specify","custom","parameters","(","n_gpus","etc","...",")",".","If","you","input","erroneous","parameters",",","an","error","will","print","all","parameters","available","for","you","."]
docstr: Updates sbatch submission file parameters
  
  Parameters
  ----------
  See slurm documentation for most parameters.
  Most useful parameters are: time, mem, gpus_per_node, cpus_per_task, partition
  Below are the parameters that differ from slurm documentation:
  
  signal_delay_s: int
      delay between the kill signal and the actual kill of the slurm job.
  setup: list
      a list of command to run in sbatch befure running srun
  array_parallelism: int
      number of map tasks that will be executed in parallel
  
  Raises
  ------
  ValueError
      In case an erroneous keyword argument is added, a list of all eligible parameters
      is printed, with their default values
  
  Note
  ----
  Best practice (as far as Quip is concerned): cpus_per_task=2x (number of data workers + gpus_per_task)
  You can use cpus_per_gpu=2 (requires using gpus_per_task and not gpus_per_node)
docstr_tok: ["Updates","sbatch","submission","file","parameters","Parameters","--","--","--","--","--","See","slurm","documentation","for","most","parameters",".","Most","useful","parameters","are",":","time",",","mem",",","gpus_per_node",",","cpus_per_task",",","partition","Below","are","the","parameters","that","differ","from","slurm","documentation",":","signal_delay_s",":","int","delay","between","the","kill","signal","and","the","actual","kill","of","the","slurm","job",".","setup",":","list","a","list","of","command","to","run","in","sbatch","befure","running","srun","array_parallelism",":","int","number","of","map","tasks","that","will","be","executed","in","parallel","Raises","--","--","--","ValueError","In","case","an","erroneous","keyword","argument","is","added",",","a","list","of","all","eligible","parameters","is","printed",",","with","their","default","values","Note","--","--","Best","practice","(","as","far","as","Quip","is","concerned",")",":","cpus_per_task=2x","(","number","of","data","workers","+","gpus_per_task",")","You","can","use","cpus_per_gpu=2","(","requires","using","gpus_per_task","and","not","gpus_per_node",")"]
code: def _internal_update_parameters(self, **kwargs: tp.Any) -> None:
      """Updates sbatch submission file parameters
  
      Parameters
      ----------
      See slurm documentation for most parameters.
      Most useful parameters are: time, mem, gpus_per_node, cpus_per_task, partition
      Below are the parameters that differ from slurm documentation:
  
      signal_delay_s: int
          delay between the kill signal and the actual kill of the slurm job.
      setup: list
          a list of command to run in sbatch befure running srun
      array_parallelism: int
          number of map tasks that will be executed in parallel
  
      Raises
      ------
      ValueError
          In case an erroneous keyword argument is added, a list of all eligible parameters
          is printed, with their default values
  
      Note
      ----
      Best practice (as far as Quip is concerned): cpus_per_task=2x (number of data workers + gpus_per_task)
      You can use cpus_per_gpu=2 (requires using gpus_per_task and not gpus_per_node)
      """
      defaults = _get_default_parameters()
      in_valid_parameters = sorted(set(kwargs) - set(defaults))
      if in_valid_parameters:
          string = "\n  - ".join(
              f"{x} (default: {repr(y)})"
              for x, y in sorted(defaults.items())
          )
          raise ValueError(
              f"Unavailable parameter(s): {in_valid_parameters}\nValid parameters are:\n  - {string}"
          )
      # check that new parameters are correct
      _make_sbatch_string(
          command="nothing to do", folder=self.folder, **kwargs
      )
      super()._internal_update_parameters(**kwargs)
  
imports: import functools
  import inspect
  import os
  import re
  import shlex
  import shutil
  import subprocess
  import sys
  import typing
  import uuid
  import warnings
  from pathlib import Path
  from ..core import core, job_environment, logger, utils
label_desc: Use image node_id 5 for calling the SlurmExecutor obj's underlying member method code with example usage: obj._internal_update_parameters() without return types