class_docstr: Run prepared procedures (DataLad scripts) on a dataset
  
  *Concept*
  
  A "procedure" is an algorithm with the purpose to process a dataset in a
  particular way. Procedures can be useful in a wide range of scenarios,
  like adjusting dataset configuration in a uniform fashion, populating
  a dataset with particular content, or automating other routine tasks,
  such as synchronizing dataset content with certain siblings.
  
  Implementations of some procedures are shipped together with DataLad,
  but additional procedures can be provided by 1) any DataLad extension,
  2) any (sub-)dataset, 3) a local user, or 4) a local system administrator.
  DataLad will look for procedures in the following locations and order:
  
  Directories identified by the configuration settings
  
  - 'datalad.locations.user-procedures' (determined by
    platformdirs.user_config_dir; defaults to '$HOME/.config/datalad/procedures'
    on GNU/Linux systems)
  - 'datalad.locations.system-procedures' (determined by
    platformdirs.site_config_dir; defaults to '/etc/xdg/datalad/procedures' on
    GNU/Linux systems)
  - 'datalad.locations.dataset-procedures'
  
  and subsequently in the 'resources/procedures/' directories of any
  installed extension, and, lastly, of the DataLad installation itself.
  
  Please note that a dataset that defines
  'datalad.locations.dataset-procedures' provides its procedures to
  any dataset it is a subdataset of. That way you can have a collection of
  such procedures in a dedicated dataset and install it as a subdataset into
  any dataset you want to use those procedures with. In case of a naming
  conflict with such a dataset hierarchy, the dataset you're calling
  run-procedures on will take precedence over its subdatasets and so on.
  
  Each configuration setting can occur multiple times to indicate multiple
  directories to be searched. If a procedure matching a given name is found
  (filename without a possible extension), the search is aborted and this
  implementation will be executed. This makes it possible for individual
  datasets, users, or machines to override externally provided procedures
  (enabling the implementation of customizable processing "hooks").
  
  
  *Procedure implementation*
  
  A procedure can be any executable. Executables must have the appropriate
  permissions and, in the case of a script, must contain an appropriate
  "shebang" line. If a procedure is not executable, but its filename ends
  with '.py', it is automatically executed by the 'python' interpreter
  (whichever version is available in the present environment). Likewise,
  procedure implementations ending on '.sh' are executed via 'bash'.
  
  Procedures can implement any argument handling, but must be capable
  of taking at least one positional argument (the absolute path to the
  dataset they shall operate on).
  
  For further customization there are two configuration settings per procedure
  available:
  
  - 'datalad.procedures.<NAME>.call-format'
    fully customizable format string to determine how to execute procedure
    NAME (see also datalad-run).
    It currently requires to include the following placeholders:
  
    - '{script}': will be replaced by the path to the procedure
    - '{ds}': will be replaced by the absolute path to the dataset the
      procedure shall operate on
    - '{args}': (not actually required) will be replaced by
      [CMD: all additional arguments passed into run-procedure after NAME CMD]
      [PY: all but the first element of `spec` if `spec` is a list or tuple PY]
      As an example the default format string for a call to a python script is:
      "python {script} {ds} {args}"
  - 'datalad.procedures.<NAME>.help'
    will be shown on `datalad run-procedure --help-proc NAME` to provide a
    description and/or usage info for procedure NAME
class_docstr_tok: ["Run","prepared","procedures","(","DataLad","scripts",")","on","a","dataset","*","Concept","*","A","``","procedure","''","is","an","algorithm","with","the","purpose","to","process","a","dataset","in","a","particular","way",".","Procedures","can","be","useful","in","a","wide","range","of","scenarios",",","like","adjusting","dataset","configuration","in","a","uniform","fashion",",","populating","a","dataset","with","particular","content",",","or","automating","other","routine","tasks",",","such","as","synchronizing","dataset","content","with","certain","siblings",".","Implementations","of","some","procedures","are","shipped","together","with","DataLad",",","but","additional","procedures","can","be","provided","by","1",")","any","DataLad","extension",",","2",")","any","(","sub-",")","dataset",",","3",")","a","local","user",",","or","4",")","a","local","system","administrator",".","DataLad","will","look","for","procedures","in","the","following","locations","and","order",":","Directories","identified","by","the","configuration","settings","-","'datalad.locations.user-procedures","'","(","determined","by","platformdirs.user_config_dir",";","defaults","to","'","$","HOME\/.config\/datalad\/procedures'","on","GNU\/Linux","systems",")","-","'datalad.locations.system-procedures","'","(","determined","by","platformdirs.site_config_dir",";","defaults","to","'\/etc\/xdg\/datalad\/procedures","'","on","GNU\/Linux","systems",")","-","'datalad.locations.dataset-procedures'","and","subsequently","in","the","'resources\/procedures\/","'","directories","of","any","installed","extension",",","and",",","lastly",",","of","the","DataLad","installation","itself",".","Please","note","that","a","dataset","that","defines","'datalad.locations.dataset-procedures","'","provides","its","procedures","to","any","dataset","it","is","a","subdataset","of",".","That","way","you","can","have","a","collection","of","such","procedures","in","a","dedicated","dataset","and","install","it","as","a","subdataset","into","any","dataset","you","want","to","use","those","procedures","with",".","In","case","of","a","naming","conflict","with","such","a","dataset","hierarchy",",","the","dataset","you","'re","calling","run-procedures","on","will","take","precedence","over","its","subdatasets","and","so","on",".","Each","configuration","setting","can","occur","multiple","times","to","indicate","multiple","directories","to","be","searched",".","If","a","procedure","matching","a","given","name","is","found","(","filename","without","a","possible","extension",")",",","the","search","is","aborted","and","this","implementation","will","be","executed",".","This","makes","it","possible","for","individual","datasets",",","users",",","or","machines","to","override","externally","provided","procedures","(","enabling","the","implementation","of","customizable","processing","``","hooks","''",")",".","*","Procedure","implementation","*","A","procedure","can","be","any","executable",".","Executables","must","have","the","appropriate","permissions","and",",","in","the","case","of","a","script",",","must","contain","an","appropriate","''","shebang","''","line",".","If","a","procedure","is","not","executable",",","but","its","filename","ends","with","'.py","'",",","it","is","automatically","executed","by","the","'python","'","interpreter","(","whichever","version","is","available","in","the","present","environment",")",".","Likewise",",","procedure","implementations","ending","on","'.sh","'","are","executed","via","'bash","'",".","Procedures","can","implement","any","argument","handling",",","but","must","be","capable","of","taking","at","least","one","positional","argument","(","the","absolute","path","to","the","dataset","they","shall","operate","on",")",".","For","further","customization","there","are","two","configuration","settings","per","procedure","available",":","-","'datalad.procedures.","<","NAME",">",".call-format'","fully","customizable","format","string","to","determine","how","to","execute","procedure","NAME","(","see","also","datalad-run",")",".","It","currently","requires","to","include","the","following","placeholders",":","-","'","{","script","}","'",":","will","be","replaced","by","the","path","to","the","procedure","-","'","{","ds","}","'",":","will","be","replaced","by","the","absolute","path","to","the","dataset","the","procedure","shall","operate","on","-","'","{","args","}","'",":","(","not","actually","required",")","will","be","replaced","by","[","CMD",":","all","additional","arguments","passed","into","run-procedure","after","NAME","CMD","]","[","PY",":","all","but","the","first","element","of","`","spec","`","if","`","spec","`","is","a","list","or","tuple","PY","]","As","an","example","the","default","format","string","for","a","call","to","a","python","script","is",":","``","python","{","script","}","{","ds","}","{","args","}","''","-","'datalad.procedures.","<","NAME",">",".help'","will","be","shown","on","`","datalad","run-procedure","--","help-proc","NAME","`","to","provide","a","description","and\/or","usage","info","for","procedure","NAME"]
code: def __call__(
      spec=None, *, dataset=None, discover=False, help_proc=False
  ):
      if not spec and not discover:
          raise InsufficientArgumentsError(
              "requires at least a procedure name"
          )
      if help_proc and not spec:
          raise InsufficientArgumentsError("requires a procedure name")
  
      try:
          ds = require_dataset(
              dataset, check_installed=False, purpose="run a procedure"
          )
      except NoDatasetFound:
          ds = None
  
      if discover:
          # specific path of procedures that were already reported
          reported = set()
          # specific names of procedure for which an active one has been
          # found
          active = set()
          for (
              m,
              cmd_name,
              cmd_tmpl,
              cmd_help,
          ) in _get_procedure_implementation("*", ds=ds):
              if m in reported:
                  continue
              ex = _guess_exec(m)
              # configured template (call-format string) takes precedence:
              if cmd_tmpl:
                  ex["template"] = cmd_tmpl
              if ex["state"] is None:
                  # doesn't seem like a match
                  lgr.debug(
                      "%s does not look like a procedure, ignored.", m
                  )
                  continue
              state = (
                  "overridden" if cmd_name in active else ex["state"]
              )
              message = ex["type"] if ex["type"] else "unknown type"
              message += (
                  " ({})".format(state) if state != "executable" else ""
              )
              res = get_status_dict(
                  action="discover_procedure",
                  path=m,
                  type="file",
                  logger=lgr,
                  refds=ds.path if ds else None,
                  status="ok",
                  state=state,
                  procedure_name=cmd_name,
                  procedure_type=ex["type"],
                  procedure_callfmt=ex["template"],
                  procedure_help=cmd_help,
                  message=message,
              )
              reported.add(m)
              if state == "executable":
                  active.add(cmd_name)
              yield res
          return
  
      if isinstance(spec, dict):
          # Skip getting procedure implementation if called with a
          # dictionary (presumably coming from --discover)
          procedure_file = spec["path"]
          cmd_name = spec["procedure_name"]
          cmd_tmpl = spec["procedure_callfmt"]
          cmd_help = spec["procedure_help"]
  
          name = cmd_name
          args = []
  
      else:
          if not isinstance(spec, (tuple, list)):
              # maybe coming from config
              spec = split_cmdline(spec)
          name = spec[0]
          args = spec[1:]
  
          try:
              # get the first match an run with it
              procedure_file, cmd_name, cmd_tmpl, cmd_help = next(
                  _get_procedure_implementation(name, ds=ds)
              )
          except StopIteration:
              raise ValueError(
                  "Cannot find procedure with name '%s'" % name
              )
  
      ex = _guess_exec(procedure_file)
      # configured template (call-format string) takes precedence:
      if cmd_tmpl:
          ex["template"] = cmd_tmpl
  
      if help_proc:
          if cmd_help:
              res = get_status_dict(
                  action="procedure_help",
                  path=procedure_file,
                  type="file",
                  logger=lgr,
                  refds=ds.path if ds else None,
                  status="ok",
                  state=ex["state"],
                  procedure_name=cmd_name,
                  procedure_type=ex["type"],
                  procedure_callfmt=ex["template"],
                  message=cmd_help,
              )
          else:
              res = get_status_dict(
                  action="procedure_help",
                  path=procedure_file,
                  type="file",
                  logger=lgr,
                  refds=ds.path if ds else None,
                  status="impossible",
                  state=ex["state"],
                  procedure_name=cmd_name,
                  procedure_type=ex["type"],
                  procedure_callfmt=ex["template"],
                  message="No help available for '%s'" % name,
              )
  
          yield res
          return
  
      if not ex["template"]:
          raise ValueError(
              "No idea how to execute procedure %s. "
              "Missing 'execute' permissions?" % procedure_file
          )
  
      cmd = ex["template"].format(
          script=guard_for_format(quote_cmdlinearg(procedure_file)),
          ds=guard_for_format(quote_cmdlinearg(ds.path)) if ds else "",
          args=join_cmdline(args) if args else "",
      )
      lgr.info("Running procedure %s", name)
      lgr.debug("Full procedure command: %r", cmd)
      for r in Run.__call__(
          cmd=cmd,
          dataset=ds,
          explicit=True,
          inputs=None,
          outputs=None,
          # pass through here
          on_failure="ignore",
          return_type="generator",
          result_renderer="disabled",
      ):
          yield r
  
      if ds:
          # the procedure ran and we have to anticipate that it might have
          # changed the dataset config, so we need to trigger an unforced
          # reload.
          # we have to do this despite "being done here", because
          # run_procedure() runs in the same process and reuses dataset (config
          # manager) instances, and the next interaction with a dataset should
          # be able to count on an up-to-date config
          ds.config.reload()
  
imports: import logging
  import os
  import os.path
  import stat
  import sys
  from argparse import REMAINDER
  from glob import iglob
  import datalad.support.ansi_colors
  from datalad import cfg
  from datalad.core.local.run import Run
  from datalad.distribution.dataset import Dataset, EnsureDataset, datasetmethod, require_dataset
  from datalad.interface.base import Interface, build_doc, eval_results
  from datalad.interface.results import get_status_dict
  from datalad.support.constraints import EnsureNone
  from datalad.support.exceptions import InsufficientArgumentsError, NoDatasetFound
  from datalad.support.param import Parameter
  from datalad.utils import ensure_list, guard_for_format, join_cmdline, quote_cmdlinearg, split_cmdline
label_desc: Use image node_id 1 for calling the RunProcedure obj's underlying member method code with example usage: obj.__call__(spec) without return types