prompt: We are currently solving the following issue within our repository. Here is the issue text:
  --- BEGIN ISSUE ---
  [Feature Request] [Submitit-Plugin] (Potentially a bug) Impossible to set certain flags in submitit launcher
  # 🚀 Feature Request
  
  <!-- A clear and concise description of the feature you are requesting -->
  
  ## Motivation
  
  This could be read as a feature request or a bug report, I'm not sure how you want to consider it, I'm going with feature request. Currently theres some flags that you cannot set in the submitit launcher. I practice "gpu centric" scheduling, so I like to specify `mem_per_gpu` and `cpus_per_gpu` and then I can just use `gpus_per_task` to always get the optimal settings. 
  
  For example on "mystery cluster" (you know the one), we can use 10 cpus and 64GB RAM per GPU. These settings allow me to only have to change `gpus_per_task` for example if I need to have 2 GPUs for 2 different models, then I'll automatically get 20 cpus and 128GB RAM for each of the tasks without having to change all of the settings. I've already PRed stuff related to this into submitit.
  
  The problem occurs when you try to set, via additional parameters, something like `mem_per_gpu`. You can't set `mem` and `mem_per_gpu`, slurm just crashes when you do that. Similarly, if you try to set `cpus_per_gpu` via `additional_parameters` you'll wind up setting that in addition to `cpus_per_task`.
  
  ## Pitch
  
  I tried making a simple patch that fixes this, but it hits issues with the `AutoExecutor` which I never updated in submitit to be aware of the options which conflict with each other. I think in general auto is missing some of the recent work in submitit, and it feels like a semi-hacky workaround to me. 
  
  Submitit already has pretty good validation logic so my pitch is to (1) let submitit handle everything by calling the correct executor for the job instead of using Auto. This is going to require a revamp of how the parameters are named/passed however and will likely be a breaking change to the API. (2) It would be nice if we could allow people to pass whatever parameters are supported by submitit without needing to update the hydra schema each time. These are my two major goals, so they sound reasonable/feasible? Point (2) may not be possible I guess.
  
  Other options:
  
  * Use auto but try to include our own validation logic (needs exploration), I don't like this because it requires us to duplicate logic that submitit has already and also it may not even be possible depending on what Auto decides to do (looked iffy on my quick pass)
  * Punt this to submitit to revamp the auto executor to do the right stuff and be more flexible. I don't like this because I think the auto executor is more of a convenience hack and more complex/flexible code would be calling the executors directly (this is what I was doing previously with my own submitit hydra wrapper thing). I also think its going to be a more more complex PR.
  
  **Are you willing to open a pull request?** (See [CONTRIBUTING](../../CONTRIBUTING.md))
  
  Yes, but I want feedback first on the best way to go about it
  
  ## Additional context
  Add any other context or screenshots about the feature request here.
  
  
  --- END ISSUE ---
  
  Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
  --- BEGIN FILES ---
  Path: `plugins/hydra_submitit_launcher/hydra_plugins/hydra_submitit_launcher/config.py`
  Content:
  ```
  1 # Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
  2 from dataclasses import dataclass, field
  3 from typing import Any, Dict, List, Optional
  4 
  5 from hydra.core.config_store import ConfigStore
  6 
  7 
  8 @dataclass
  9 class BaseQueueConf:
  10     """Configuration shared by all executors"""
  11 
  12     submitit_folder: str = "${hydra.sweep.dir}/.submitit/%j"
  13 
  14     # maximum time for the job in minutes
  15     timeout_min: int = 60
  16     # number of cpus to use for each task
  17     cpus_per_task: int = 1
  18     # number of gpus to use on each node
  19     gpus_per_node: int = 0
  20     # number of tasks to spawn on each node
  21     tasks_per_node: int = 1
  22     # memory to reserve for the job on each node (in GB)
  23     mem_gb: int = 4
  24     # number of nodes to use for the job
  25     nodes: int = 1
  26     # name of the job
  27     name: str = "${hydra.job.name}"
  28 
  29 
  30 @dataclass
  31 class SlurmQueueConf(BaseQueueConf):
  32     """Slurm configuration overrides and specific parameters"""
  33 
  34     _target_: str = (
  35         "hydra_plugins.hydra_submitit_launcher.submitit_launcher.SlurmLauncher"
  36     )
  37 
  38     # Params are used to configure sbatch, for more info check:
  39     # https://github.com/facebookincubator/submitit/blob/master/submitit/slurm/slurm.py
  40 
  41     # Following parameters are slurm specific
  42     # More information: https://slurm.schedmd.com/sbatch.html
  43     #
  44     # slurm partition to use on the cluster
  45     partition: Optional[str] = None
  46     comment: Optional[str] = None
  47     constraint: Optional[str] = None
  48     exclude: Optional[str] = None
  49 
  50     # Following parameters are submitit specifics
  51     #
  52     # USR1 signal delay before timeout
  53     signal_delay_s: int = 120
  54     # Maximum number of retries on job timeout.
  55     # Change this only after you confirmed your code can handle re-submission
  56     # by properly resuming from the latest stored checkpoint.
  57     # check the following for more info on slurm_max_num_timeout
  58     # https://github.com/facebookincubator/submitit/blob/master/docs/checkpointing.md
  59     max_num_timeout: int = 0
  60     # Useful to add parameters which are not currently available in the plugin.
  61     # Eg: {"mail-user": "blublu@fb.com", "mail-type": "BEGIN"}
  62     additional_parameters: Dict[str, Any] = field(default_factory=dict)
  63     # Maximum number of jobs running in parallel
  64     array_parallelism: int = 256
  65     # A list of commands to run in sbatch befure running srun
  66     setup: Optional[List[str]] = None
  67 
  68 
  69 @dataclass
  70 class LocalQueueConf(BaseQueueConf):
  71     _target_: str = (
  72         "hydra_plugins.hydra_submitit_launcher.submitit_launcher.LocalLauncher"
  73     )
  74 
  75 
  76 # finally, register two different choices:
  77 ConfigStore.instance().store(
  78     group="hydra/launcher",
  79     name="submitit_local",
  80     node=LocalQueueConf(),
  81     provider="submitit_launcher",
  82 )
  83 
  84 
  85 ConfigStore.instance().store(
  86     group="hydra/launcher",
  87     name="submitit_slurm",
  88     node=SlurmQueueConf(),
  89     provider="submitit_launcher",
  90 )
  91
  ```
  --- END FILES ---
  
  Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.
  
  Here is an example:
  
  ```diff
  diff --git a/examples/server_async.py b/examples/server_async.py
  --- a/examples/server_async.py
  +++ b/examples/server_async.py
  @@ -313,4 +313,4 @@
   
   
   if __name__ == "__main__":
  -    asyncio.run(run_async_server("."), debug=True)
  +    asyncio.run(run_async_server(), debug=True)
  diff --git a/examples/server_sync.py b/examples/server_sync.py
  --- a/examples/server_sync.py
  +++ b/examples/server_sync.py
  @@ -313,5 +313,5 @@
   
   
   if __name__ == "__main__":
  -    server = run_sync_server(".")
  +    server = run_sync_server()
       server.shutdown()
  
  ```
  
golden_diff: diff --git a/plugins/hydra_submitit_launcher/hydra_plugins/hydra_submitit_launcher/config.py b/plugins/hydra_submitit_launcher/hydra_plugins/hydra_submitit_launcher/config.py
  --- a/plugins/hydra_submitit_launcher/hydra_plugins/hydra_submitit_launcher/config.py
  +++ b/plugins/hydra_submitit_launcher/hydra_plugins/hydra_submitit_launcher/config.py
  @@ -14,13 +14,13 @@
       # maximum time for the job in minutes
       timeout_min: int = 60
       # number of cpus to use for each task
  -    cpus_per_task: int = 1
  +    cpus_per_task: Optional[int] = None
       # number of gpus to use on each node
  -    gpus_per_node: int = 0
  +    gpus_per_node: Optional[int] = None
       # number of tasks to spawn on each node
       tasks_per_node: int = 1
       # memory to reserve for the job on each node (in GB)
  -    mem_gb: int = 4
  +    mem_gb: Optional[int] = None
       # number of nodes to use for the job
       nodes: int = 1
       # name of the job
  @@ -46,6 +46,10 @@
       comment: Optional[str] = None
       constraint: Optional[str] = None
       exclude: Optional[str] = None
  +    cpus_per_gpu: Optional[int] = None
  +    gpus_per_task: Optional[int] = None
  +    mem_per_gpu: Optional[str] = None
  +    mem_per_cpu: Optional[str] = None
   
       # Following parameters are submitit specifics
       #
  
verification_info: {"golden_diff": "diff --git a/plugins/hydra_submitit_launcher/hydra_plugins/hydra_submitit_launcher/config.py b/plugins/hydra_submitit_launcher/hydra_plugins/hydra_submitit_launcher/config.py\n--- a/plugins/hydra_submitit_launcher/hydra_plugins/hydra_submitit_launcher/config.py\n+++ b/plugins/hydra_submitit_launcher/hydra_plugins/hydra_submitit_launcher/config.py\n@@ -14,13 +14,13 @@\n     # maximum time for the job in minutes\n     timeout_min: int = 60\n     # number of cpus to use for each task\n-    cpus_per_task: int = 1\n+    cpus_per_task: Optional[int] = None\n     # number of gpus to use on each node\n-    gpus_per_node: int = 0\n+    gpus_per_node: Optional[int] = None\n     # number of tasks to spawn on each node\n     tasks_per_node: int = 1\n     # memory to reserve for the job on each node (in GB)\n-    mem_gb: int = 4\n+    mem_gb: Optional[int] = None\n     # number of nodes to use for the job\n     nodes: int = 1\n     # name of the job\n@@ -46,6 +46,10 @@\n     comment: Optional[str] = None\n     constraint: Optional[str] = None\n     exclude: Optional[str] = None\n+    cpus_per_gpu: Optional[int] = None\n+    gpus_per_task: Optional[int] = None\n+    mem_per_gpu: Optional[str] = None\n+    mem_per_cpu: Optional[str] = None\n \n     # Following parameters are submitit specifics\n     #\n", "issue": "[Feature Request] [Submitit-Plugin] (Potentially a bug) Impossible to set certain flags in submitit launcher\n# \ud83d\ude80 Feature Request\r\n\r\n<!-- A clear and concise description of the feature you are requesting -->\r\n\r\n## Motivation\r\n\r\nThis could be read as a feature request or a bug report, I'm not sure how you want to consider it, I'm going with feature request. Currently theres some flags that you cannot set in the submitit launcher. I practice \"gpu centric\" scheduling, so I like to specify `mem_per_gpu` and `cpus_per_gpu` and then I can just use `gpus_per_task` to always get the optimal settings. \r\n\r\nFor example on \"mystery cluster\" (you know the one), we can use 10 cpus and 64GB RAM per GPU. These settings allow me to only have to change `gpus_per_task` for example if I need to have 2 GPUs for 2 different models, then I'll automatically get 20 cpus and 128GB RAM for each of the tasks without having to change all of the settings. I've already PRed stuff related to this into submitit.\r\n\r\nThe problem occurs when you try to set, via additional parameters, something like `mem_per_gpu`. You can't set `mem` and `mem_per_gpu`, slurm just crashes when you do that. Similarly, if you try to set `cpus_per_gpu` via `additional_parameters` you'll wind up setting that in addition to `cpus_per_task`.\r\n\r\n## Pitch\r\n\r\nI tried making a simple patch that fixes this, but it hits issues with the `AutoExecutor` which I never updated in submitit to be aware of the options which conflict with each other. I think in general auto is missing some of the recent work in submitit, and it feels like a semi-hacky workaround to me. \r\n\r\nSubmitit already has pretty good validation logic so my pitch is to (1) let submitit handle everything by calling the correct executor for the job instead of using Auto. This is going to require a revamp of how the parameters are named/passed however and will likely be a breaking change to the API. (2) It would be nice if we could allow people to pass whatever parameters are supported by submitit without needing to update the hydra schema each time. These are my two major goals, so they sound reasonable/feasible? Point (2) may not be possible I guess.\r\n\r\nOther options:\r\n\r\n* Use auto but try to include our own validation logic (needs exploration), I don't like this because it requires us to duplicate logic that submitit has already and also it may not even be possible depending on what Auto decides to do (looked iffy on my quick pass)\r\n* Punt this to submitit to revamp the auto executor to do the right stuff and be more flexible. I don't like this because I think the auto executor is more of a convenience hack and more complex/flexible code would be calling the executors directly (this is what I was doing previously with my own submitit hydra wrapper thing). I also think its going to be a more more complex PR.\r\n\r\n**Are you willing to open a pull request?** (See [CONTRIBUTING](../../CONTRIBUTING.md))\r\n\r\nYes, but I want feedback first on the best way to go about it\r\n\r\n## Additional context\r\nAdd any other context or screenshots about the feature request here.\r\n\n", "before_files": [{"content": "# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved\nfrom dataclasses import dataclass, field\nfrom typing import Any, Dict, List, Optional\n\nfrom hydra.core.config_store import ConfigStore\n\n\n@dataclass\nclass BaseQueueConf:\n    \"\"\"Configuration shared by all executors\"\"\"\n\n    submitit_folder: str = \"${hydra.sweep.dir}/.submitit/%j\"\n\n    # maximum time for the job in minutes\n    timeout_min: int = 60\n    # number of cpus to use for each task\n    cpus_per_task: int = 1\n    # number of gpus to use on each node\n    gpus_per_node: int = 0\n    # number of tasks to spawn on each node\n    tasks_per_node: int = 1\n    # memory to reserve for the job on each node (in GB)\n    mem_gb: int = 4\n    # number of nodes to use for the job\n    nodes: int = 1\n    # name of the job\n    name: str = \"${hydra.job.name}\"\n\n\n@dataclass\nclass SlurmQueueConf(BaseQueueConf):\n    \"\"\"Slurm configuration overrides and specific parameters\"\"\"\n\n    _target_: str = (\n        \"hydra_plugins.hydra_submitit_launcher.submitit_launcher.SlurmLauncher\"\n    )\n\n    # Params are used to configure sbatch, for more info check:\n    # https://github.com/facebookincubator/submitit/blob/master/submitit/slurm/slurm.py\n\n    # Following parameters are slurm specific\n    # More information: https://slurm.schedmd.com/sbatch.html\n    #\n    # slurm partition to use on the cluster\n    partition: Optional[str] = None\n    comment: Optional[str] = None\n    constraint: Optional[str] = None\n    exclude: Optional[str] = None\n\n    # Following parameters are submitit specifics\n    #\n    # USR1 signal delay before timeout\n    signal_delay_s: int = 120\n    # Maximum number of retries on job timeout.\n    # Change this only after you confirmed your code can handle re-submission\n    # by properly resuming from the latest stored checkpoint.\n    # check the following for more info on slurm_max_num_timeout\n    # https://github.com/facebookincubator/submitit/blob/master/docs/checkpointing.md\n    max_num_timeout: int = 0\n    # Useful to add parameters which are not currently available in the plugin.\n    # Eg: {\"mail-user\": \"blublu@fb.com\", \"mail-type\": \"BEGIN\"}\n    additional_parameters: Dict[str, Any] = field(default_factory=dict)\n    # Maximum number of jobs running in parallel\n    array_parallelism: int = 256\n    # A list of commands to run in sbatch befure running srun\n    setup: Optional[List[str]] = None\n\n\n@dataclass\nclass LocalQueueConf(BaseQueueConf):\n    _target_: str = (\n        \"hydra_plugins.hydra_submitit_launcher.submitit_launcher.LocalLauncher\"\n    )\n\n\n# finally, register two different choices:\nConfigStore.instance().store(\n    group=\"hydra/launcher\",\n    name=\"submitit_local\",\n    node=LocalQueueConf(),\n    provider=\"submitit_launcher\",\n)\n\n\nConfigStore.instance().store(\n    group=\"hydra/launcher\",\n    name=\"submitit_slurm\",\n    node=SlurmQueueConf(),\n    provider=\"submitit_launcher\",\n)\n", "path": "plugins/hydra_submitit_launcher/hydra_plugins/hydra_submitit_launcher/config.py"}], "after_files": [{"content": "# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved\nfrom dataclasses import dataclass, field\nfrom typing import Any, Dict, List, Optional\n\nfrom hydra.core.config_store import ConfigStore\n\n\n@dataclass\nclass BaseQueueConf:\n    \"\"\"Configuration shared by all executors\"\"\"\n\n    submitit_folder: str = \"${hydra.sweep.dir}/.submitit/%j\"\n\n    # maximum time for the job in minutes\n    timeout_min: int = 60\n    # number of cpus to use for each task\n    cpus_per_task: Optional[int] = None\n    # number of gpus to use on each node\n    gpus_per_node: Optional[int] = None\n    # number of tasks to spawn on each node\n    tasks_per_node: int = 1\n    # memory to reserve for the job on each node (in GB)\n    mem_gb: Optional[int] = None\n    # number of nodes to use for the job\n    nodes: int = 1\n    # name of the job\n    name: str = \"${hydra.job.name}\"\n\n\n@dataclass\nclass SlurmQueueConf(BaseQueueConf):\n    \"\"\"Slurm configuration overrides and specific parameters\"\"\"\n\n    _target_: str = (\n        \"hydra_plugins.hydra_submitit_launcher.submitit_launcher.SlurmLauncher\"\n    )\n\n    # Params are used to configure sbatch, for more info check:\n    # https://github.com/facebookincubator/submitit/blob/master/submitit/slurm/slurm.py\n\n    # Following parameters are slurm specific\n    # More information: https://slurm.schedmd.com/sbatch.html\n    #\n    # slurm partition to use on the cluster\n    partition: Optional[str] = None\n    comment: Optional[str] = None\n    constraint: Optional[str] = None\n    exclude: Optional[str] = None\n    cpus_per_gpu: Optional[int] = None\n    gpus_per_task: Optional[int] = None\n    mem_per_gpu: Optional[str] = None\n    mem_per_cpu: Optional[str] = None\n\n    # Following parameters are submitit specifics\n    #\n    # USR1 signal delay before timeout\n    signal_delay_s: int = 120\n    # Maximum number of retries on job timeout.\n    # Change this only after you confirmed your code can handle re-submission\n    # by properly resuming from the latest stored checkpoint.\n    # check the following for more info on slurm_max_num_timeout\n    # https://github.com/facebookincubator/submitit/blob/master/docs/checkpointing.md\n    max_num_timeout: int = 0\n    # Useful to add parameters which are not currently available in the plugin.\n    # Eg: {\"mail-user\": \"blublu@fb.com\", \"mail-type\": \"BEGIN\"}\n    additional_parameters: Dict[str, Any] = field(default_factory=dict)\n    # Maximum number of jobs running in parallel\n    array_parallelism: int = 256\n    # A list of commands to run in sbatch befure running srun\n    setup: Optional[List[str]] = None\n\n\n@dataclass\nclass LocalQueueConf(BaseQueueConf):\n    _target_: str = (\n        \"hydra_plugins.hydra_submitit_launcher.submitit_launcher.LocalLauncher\"\n    )\n\n\n# finally, register two different choices:\nConfigStore.instance().store(\n    group=\"hydra/launcher\",\n    name=\"submitit_local\",\n    node=LocalQueueConf(),\n    provider=\"submitit_launcher\",\n)\n\n\nConfigStore.instance().store(\n    group=\"hydra/launcher\",\n    name=\"submitit_slurm\",\n    node=SlurmQueueConf(),\n    provider=\"submitit_launcher\",\n)\n", "path": "plugins/hydra_submitit_launcher/hydra_plugins/hydra_submitit_launcher/config.py"}]}