We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
failed to train with remote mode
hello, everyone,
      I used the experiment examples/trials/weight_sharing/ga_squad/config_remote.yml, and I modified the machineList with my own machine, but the experiment was waiting and the log is below. Is there any suggestion?
```
[9/29/2019, 4:40:41 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:40:41 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:40:44 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:40:44 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:40:47 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:40:47 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:40:50 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:40:50 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:40:53 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:40:53 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:40:56 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:40:56 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:40:59 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:40:59 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:02 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:02 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:05 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:05 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:08 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:08 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:11 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:11 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:14 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:14 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:17 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:17 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:20 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:20 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:23 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:23 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:26 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:26 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:29 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:29 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:32 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
[9/29/2019, 4:41:32 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]
[9/29/2019, 4:41:35 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]
```

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `tools/nni_gpu_tool/gpu_metrics_collector.py`
Content:
```
1 # Copyright (c) Microsoft Corporation.
2 # Licensed under the MIT license.
3 
4 import json
5 import os
6 import subprocess
7 import sys
8 import time
9 import traceback
10 
11 from xml.dom import minidom
12 
13 def check_ready_to_run():
14     if sys.platform == 'win32':
15         pgrep_output = subprocess.check_output(
16             'wmic process where "CommandLine like \'%nni_gpu_tool.gpu_metrics_collector%\' and name like \'%python%\'" get processId')
17         pidList = pgrep_output.decode("utf-8").strip().split()
18         pidList.pop(0) # remove the key word 'ProcessId'
19         pidList = list(map(int, pidList))
20         pidList.remove(os.getpid())
21         return not pidList
22     else:
23         pgrep_output = subprocess.check_output('pgrep -fxu "$(whoami)" \'python3 -m nni_gpu_tool.gpu_metrics_collector\'', shell=True)
24         pidList = []
25         for pid in pgrep_output.splitlines():
26             pidList.append(int(pid))
27         pidList.remove(os.getpid())
28         return not pidList
29 
30 def main(argv):
31     metrics_output_dir = os.environ['METRIC_OUTPUT_DIR']
32     if check_ready_to_run() == False:
33         # GPU metrics collector is already running. Exit
34         exit(2)
35     cmd = 'nvidia-smi -q -x'.split()
36     while(True):
37         try:
38             smi_output = subprocess.check_output(cmd)
39         except Exception:
40             traceback.print_exc()
41             gen_empty_gpu_metric(metrics_output_dir)
42             break
43         parse_nvidia_smi_result(smi_output, metrics_output_dir)
44         # TODO: change to sleep time configurable via arguments
45         time.sleep(5)
46 
47 def parse_nvidia_smi_result(smi, outputDir):
48     try:
49         old_umask = os.umask(0)
50         xmldoc = minidom.parseString(smi)
51         gpuList = xmldoc.getElementsByTagName('gpu')
52         with open(os.path.join(outputDir, "gpu_metrics"), 'a') as outputFile:
53             outPut = {}
54             outPut["Timestamp"] = time.asctime(time.localtime())
55             outPut["gpuCount"] = len(gpuList)
56             outPut["gpuInfos"] = []
57             for gpuIndex, gpu in enumerate(gpuList):
58                 gpuInfo = {}
59                 gpuInfo['index'] = gpuIndex
60                 gpuInfo['gpuUtil'] = gpu.getElementsByTagName('utilization')[0]\
61                     .getElementsByTagName('gpu_util')[0]\
62                     .childNodes[0].data.replace("%", "").strip()
63                 gpuInfo['gpuMemUtil'] = gpu.getElementsByTagName('utilization')[0]\
64                     .getElementsByTagName('memory_util')[0]\
65                     .childNodes[0].data.replace("%", "").strip()
66                 processes = gpu.getElementsByTagName('processes')
67                 runningProNumber = len(processes[0].getElementsByTagName('process_info'))
68                 gpuInfo['activeProcessNum'] = runningProNumber
69 
70                 outPut["gpuInfos"].append(gpuInfo)
71             print(outPut)
72             outputFile.write("{}\n".format(json.dumps(outPut, sort_keys=True)))
73             outputFile.flush();
74     except:
75         # e_info = sys.exc_info()
76         print('xmldoc paring error')
77     finally:
78         os.umask(old_umask)
79 
80 def gen_empty_gpu_metric(outputDir):
81     try:
82         old_umask = os.umask(0)
83         with open(os.path.join(outputDir, "gpu_metrics"), 'a') as outputFile:
84             outPut = {}
85             outPut["Timestamp"] = time.asctime(time.localtime())
86             outPut["gpuCount"] = 0
87             outPut["gpuInfos"] = []
88             print(outPut)
89             outputFile.write("{}\n".format(json.dumps(outPut, sort_keys=True)))
90             outputFile.flush()
91     except Exception:
92         traceback.print_exc()
93     finally:
94         os.umask(old_umask)
95 
96 
97 if __name__ == "__main__":
98     main(sys.argv[1:])
99
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/tools/nni_gpu_tool/gpu_metrics_collector.py b/tools/nni_gpu_tool/gpu_metrics_collector.py
--- a/tools/nni_gpu_tool/gpu_metrics_collector.py
+++ b/tools/nni_gpu_tool/gpu_metrics_collector.py
@@ -10,27 +10,31 @@
 
 from xml.dom import minidom
 
+
 def check_ready_to_run():
     if sys.platform == 'win32':
         pgrep_output = subprocess.check_output(
             'wmic process where "CommandLine like \'%nni_gpu_tool.gpu_metrics_collector%\' and name like \'%python%\'" get processId')
         pidList = pgrep_output.decode("utf-8").strip().split()
-        pidList.pop(0) # remove the key word 'ProcessId'
+        pidList.pop(0)  # remove the key word 'ProcessId'
         pidList = list(map(int, pidList))
         pidList.remove(os.getpid())
         return not pidList
     else:
-        pgrep_output = subprocess.check_output('pgrep -fxu "$(whoami)" \'python3 -m nni_gpu_tool.gpu_metrics_collector\'', shell=True)
+        pgrep_output = subprocess.check_output('pgrep -afu "$(whoami)" \'python3 -m nni_gpu_tool.gpu_metrics_collector\'', shell=True)
         pidList = []
         for pid in pgrep_output.splitlines():
-            pidList.append(int(pid))
-        pidList.remove(os.getpid())
+            pid = pid.decode()
+            if "pgrep " in pid or pid.startswith('%s ' % os.getpid()) or pid.startswith('%s ' % os.getppid()):
+                continue
+            pidList.append(pid)
         return not pidList
 
+
 def main(argv):
     metrics_output_dir = os.environ['METRIC_OUTPUT_DIR']
     if check_ready_to_run() == False:
-        # GPU metrics collector is already running. Exit
+        print("GPU metrics collector is already running. exiting...")
         exit(2)
     cmd = 'nvidia-smi -q -x'.split()
     while(True):
@@ -44,6 +48,7 @@
         # TODO: change to sleep time configurable via arguments
         time.sleep(5)
 
+
 def parse_nvidia_smi_result(smi, outputDir):
     try:
         old_umask = os.umask(0)
@@ -70,13 +75,14 @@
                 outPut["gpuInfos"].append(gpuInfo)
             print(outPut)
             outputFile.write("{}\n".format(json.dumps(outPut, sort_keys=True)))
-            outputFile.flush();
-    except:
+            outputFile.flush()
+    except Exception as error:
         # e_info = sys.exc_info()
-        print('xmldoc paring error')
+        print('gpu_metrics_collector error: %s' % error)
     finally:
         os.umask(old_umask)
 
+
 def gen_empty_gpu_metric(outputDir):
     try:
         old_umask = os.umask(0)

{"golden_diff": "diff --git a/tools/nni_gpu_tool/gpu_metrics_collector.py b/tools/nni_gpu_tool/gpu_metrics_collector.py\n--- a/tools/nni_gpu_tool/gpu_metrics_collector.py\n+++ b/tools/nni_gpu_tool/gpu_metrics_collector.py\n@@ -10,27 +10,31 @@\n \n from xml.dom import minidom\n \n+\n def check_ready_to_run():\n     if sys.platform == 'win32':\n         pgrep_output = subprocess.check_output(\n             'wmic process where \"CommandLine like \\'%nni_gpu_tool.gpu_metrics_collector%\\' and name like \\'%python%\\'\" get processId')\n         pidList = pgrep_output.decode(\"utf-8\").strip().split()\n-        pidList.pop(0) # remove the key word 'ProcessId'\n+        pidList.pop(0)  # remove the key word 'ProcessId'\n         pidList = list(map(int, pidList))\n         pidList.remove(os.getpid())\n         return not pidList\n     else:\n-        pgrep_output = subprocess.check_output('pgrep -fxu \"$(whoami)\" \\'python3 -m nni_gpu_tool.gpu_metrics_collector\\'', shell=True)\n+        pgrep_output = subprocess.check_output('pgrep -afu \"$(whoami)\" \\'python3 -m nni_gpu_tool.gpu_metrics_collector\\'', shell=True)\n         pidList = []\n         for pid in pgrep_output.splitlines():\n-            pidList.append(int(pid))\n-        pidList.remove(os.getpid())\n+            pid = pid.decode()\n+            if \"pgrep \" in pid or pid.startswith('%s ' % os.getpid()) or pid.startswith('%s ' % os.getppid()):\n+                continue\n+            pidList.append(pid)\n         return not pidList\n \n+\n def main(argv):\n     metrics_output_dir = os.environ['METRIC_OUTPUT_DIR']\n     if check_ready_to_run() == False:\n-        # GPU metrics collector is already running. Exit\n+        print(\"GPU metrics collector is already running. exiting...\")\n         exit(2)\n     cmd = 'nvidia-smi -q -x'.split()\n     while(True):\n@@ -44,6 +48,7 @@\n         # TODO: change to sleep time configurable via arguments\n         time.sleep(5)\n \n+\n def parse_nvidia_smi_result(smi, outputDir):\n     try:\n         old_umask = os.umask(0)\n@@ -70,13 +75,14 @@\n                 outPut[\"gpuInfos\"].append(gpuInfo)\n             print(outPut)\n             outputFile.write(\"{}\\n\".format(json.dumps(outPut, sort_keys=True)))\n-            outputFile.flush();\n-    except:\n+            outputFile.flush()\n+    except Exception as error:\n         # e_info = sys.exc_info()\n-        print('xmldoc paring error')\n+        print('gpu_metrics_collector error: %s' % error)\n     finally:\n         os.umask(old_umask)\n \n+\n def gen_empty_gpu_metric(outputDir):\n     try:\n         old_umask = os.umask(0)\n", "issue": "failed to train with remote mode\nhello, everyone,\r\n      I used the experiment examples/trials/weight_sharing/ga_squad/config_remote.yml, and I modified the machineList with my own machine, but the experiment was waiting and the log is below. Is there any suggestion?\r\n```\r\n[9/29/2019, 4:40:41 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]\r\n[9/29/2019, 4:40:41 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]\r\n[9/29/2019, 4:40:44 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]\r\n[9/29/2019, 4:40:44 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]\r\n[9/29/2019, 4:40:47 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]\r\n[9/29/2019, 4:40:47 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]\r\n[9/29/2019, 4:40:50 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]\r\n[9/29/2019, 4:40:50 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]\r\n[9/29/2019, 4:40:53 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]\r\n[9/29/2019, 4:40:53 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]\r\n[9/29/2019, 4:40:56 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]\r\n[9/29/2019, 4:40:56 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]\r\n[9/29/2019, 4:40:59 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]\r\n[9/29/2019, 4:40:59 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]\r\n[9/29/2019, 4:41:02 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]\r\n[9/29/2019, 4:41:02 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]\r\n[9/29/2019, 4:41:05 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]\r\n[9/29/2019, 4:41:05 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]\r\n[9/29/2019, 4:41:08 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]\r\n[9/29/2019, 4:41:08 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]\r\n[9/29/2019, 4:41:11 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]\r\n[9/29/2019, 4:41:11 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]\r\n[9/29/2019, 4:41:14 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]\r\n[9/29/2019, 4:41:14 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]\r\n[9/29/2019, 4:41:17 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]\r\n[9/29/2019, 4:41:17 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]\r\n[9/29/2019, 4:41:20 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]\r\n[9/29/2019, 4:41:20 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]\r\n[9/29/2019, 4:41:23 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]\r\n[9/29/2019, 4:41:23 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]\r\n[9/29/2019, 4:41:26 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]\r\n[9/29/2019, 4:41:26 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]\r\n[9/29/2019, 4:41:29 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]\r\n[9/29/2019, 4:41:29 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]\r\n[9/29/2019, 4:41:32 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]\r\n[9/29/2019, 4:41:32 PM] INFO [ 'Right now no available GPU can be allocated for trial iwzEY, will try to schedule later' ]\r\n[9/29/2019, 4:41:35 PM] WARNING [ 'Scheduler: trialJob id iwzEY, no machine can be scheduled, return TMP_NO_AVAILABLE_GPU ' ]\r\n```\n", "before_files": [{"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT license.\n\nimport json\nimport os\nimport subprocess\nimport sys\nimport time\nimport traceback\n\nfrom xml.dom import minidom\n\ndef check_ready_to_run():\n    if sys.platform == 'win32':\n        pgrep_output = subprocess.check_output(\n            'wmic process where \"CommandLine like \\'%nni_gpu_tool.gpu_metrics_collector%\\' and name like \\'%python%\\'\" get processId')\n        pidList = pgrep_output.decode(\"utf-8\").strip().split()\n        pidList.pop(0) # remove the key word 'ProcessId'\n        pidList = list(map(int, pidList))\n        pidList.remove(os.getpid())\n        return not pidList\n    else:\n        pgrep_output = subprocess.check_output('pgrep -fxu \"$(whoami)\" \\'python3 -m nni_gpu_tool.gpu_metrics_collector\\'', shell=True)\n        pidList = []\n        for pid in pgrep_output.splitlines():\n            pidList.append(int(pid))\n        pidList.remove(os.getpid())\n        return not pidList\n\ndef main(argv):\n    metrics_output_dir = os.environ['METRIC_OUTPUT_DIR']\n    if check_ready_to_run() == False:\n        # GPU metrics collector is already running. Exit\n        exit(2)\n    cmd = 'nvidia-smi -q -x'.split()\n    while(True):\n        try:\n            smi_output = subprocess.check_output(cmd)\n        except Exception:\n            traceback.print_exc()\n            gen_empty_gpu_metric(metrics_output_dir)\n            break\n        parse_nvidia_smi_result(smi_output, metrics_output_dir)\n        # TODO: change to sleep time configurable via arguments\n        time.sleep(5)\n\ndef parse_nvidia_smi_result(smi, outputDir):\n    try:\n        old_umask = os.umask(0)\n        xmldoc = minidom.parseString(smi)\n        gpuList = xmldoc.getElementsByTagName('gpu')\n        with open(os.path.join(outputDir, \"gpu_metrics\"), 'a') as outputFile:\n            outPut = {}\n            outPut[\"Timestamp\"] = time.asctime(time.localtime())\n            outPut[\"gpuCount\"] = len(gpuList)\n            outPut[\"gpuInfos\"] = []\n            for gpuIndex, gpu in enumerate(gpuList):\n                gpuInfo = {}\n                gpuInfo['index'] = gpuIndex\n                gpuInfo['gpuUtil'] = gpu.getElementsByTagName('utilization')[0]\\\n                    .getElementsByTagName('gpu_util')[0]\\\n                    .childNodes[0].data.replace(\"%\", \"\").strip()\n                gpuInfo['gpuMemUtil'] = gpu.getElementsByTagName('utilization')[0]\\\n                    .getElementsByTagName('memory_util')[0]\\\n                    .childNodes[0].data.replace(\"%\", \"\").strip()\n                processes = gpu.getElementsByTagName('processes')\n                runningProNumber = len(processes[0].getElementsByTagName('process_info'))\n                gpuInfo['activeProcessNum'] = runningProNumber\n\n                outPut[\"gpuInfos\"].append(gpuInfo)\n            print(outPut)\n            outputFile.write(\"{}\\n\".format(json.dumps(outPut, sort_keys=True)))\n            outputFile.flush();\n    except:\n        # e_info = sys.exc_info()\n        print('xmldoc paring error')\n    finally:\n        os.umask(old_umask)\n\ndef gen_empty_gpu_metric(outputDir):\n    try:\n        old_umask = os.umask(0)\n        with open(os.path.join(outputDir, \"gpu_metrics\"), 'a') as outputFile:\n            outPut = {}\n            outPut[\"Timestamp\"] = time.asctime(time.localtime())\n            outPut[\"gpuCount\"] = 0\n            outPut[\"gpuInfos\"] = []\n            print(outPut)\n            outputFile.write(\"{}\\n\".format(json.dumps(outPut, sort_keys=True)))\n            outputFile.flush()\n    except Exception:\n        traceback.print_exc()\n    finally:\n        os.umask(old_umask)\n\n\nif __name__ == \"__main__\":\n    main(sys.argv[1:])\n", "path": "tools/nni_gpu_tool/gpu_metrics_collector.py"}], "after_files": [{"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT license.\n\nimport json\nimport os\nimport subprocess\nimport sys\nimport time\nimport traceback\n\nfrom xml.dom import minidom\n\n\ndef check_ready_to_run():\n    if sys.platform == 'win32':\n        pgrep_output = subprocess.check_output(\n            'wmic process where \"CommandLine like \\'%nni_gpu_tool.gpu_metrics_collector%\\' and name like \\'%python%\\'\" get processId')\n        pidList = pgrep_output.decode(\"utf-8\").strip().split()\n        pidList.pop(0)  # remove the key word 'ProcessId'\n        pidList = list(map(int, pidList))\n        pidList.remove(os.getpid())\n        return not pidList\n    else:\n        pgrep_output = subprocess.check_output('pgrep -afu \"$(whoami)\" \\'python3 -m nni_gpu_tool.gpu_metrics_collector\\'', shell=True)\n        pidList = []\n        for pid in pgrep_output.splitlines():\n            pid = pid.decode()\n            if \"pgrep \" in pid or pid.startswith('%s ' % os.getpid()) or pid.startswith('%s ' % os.getppid()):\n                continue\n            pidList.append(pid)\n        return not pidList\n\n\ndef main(argv):\n    metrics_output_dir = os.environ['METRIC_OUTPUT_DIR']\n    if check_ready_to_run() == False:\n        print(\"GPU metrics collector is already running. exiting...\")\n        exit(2)\n    cmd = 'nvidia-smi -q -x'.split()\n    while(True):\n        try:\n            smi_output = subprocess.check_output(cmd)\n        except Exception:\n            traceback.print_exc()\n            gen_empty_gpu_metric(metrics_output_dir)\n            break\n        parse_nvidia_smi_result(smi_output, metrics_output_dir)\n        # TODO: change to sleep time configurable via arguments\n        time.sleep(5)\n\n\ndef parse_nvidia_smi_result(smi, outputDir):\n    try:\n        old_umask = os.umask(0)\n        xmldoc = minidom.parseString(smi)\n        gpuList = xmldoc.getElementsByTagName('gpu')\n        with open(os.path.join(outputDir, \"gpu_metrics\"), 'a') as outputFile:\n            outPut = {}\n            outPut[\"Timestamp\"] = time.asctime(time.localtime())\n            outPut[\"gpuCount\"] = len(gpuList)\n            outPut[\"gpuInfos\"] = []\n            for gpuIndex, gpu in enumerate(gpuList):\n                gpuInfo = {}\n                gpuInfo['index'] = gpuIndex\n                gpuInfo['gpuUtil'] = gpu.getElementsByTagName('utilization')[0]\\\n                    .getElementsByTagName('gpu_util')[0]\\\n                    .childNodes[0].data.replace(\"%\", \"\").strip()\n                gpuInfo['gpuMemUtil'] = gpu.getElementsByTagName('utilization')[0]\\\n                    .getElementsByTagName('memory_util')[0]\\\n                    .childNodes[0].data.replace(\"%\", \"\").strip()\n                processes = gpu.getElementsByTagName('processes')\n                runningProNumber = len(processes[0].getElementsByTagName('process_info'))\n                gpuInfo['activeProcessNum'] = runningProNumber\n\n                outPut[\"gpuInfos\"].append(gpuInfo)\n            print(outPut)\n            outputFile.write(\"{}\\n\".format(json.dumps(outPut, sort_keys=True)))\n            outputFile.flush()\n    except Exception as error:\n        # e_info = sys.exc_info()\n        print('gpu_metrics_collector error: %s' % error)\n    finally:\n        os.umask(old_umask)\n\n\ndef gen_empty_gpu_metric(outputDir):\n    try:\n        old_umask = os.umask(0)\n        with open(os.path.join(outputDir, \"gpu_metrics\"), 'a') as outputFile:\n            outPut = {}\n            outPut[\"Timestamp\"] = time.asctime(time.localtime())\n            outPut[\"gpuCount\"] = 0\n            outPut[\"gpuInfos\"] = []\n            print(outPut)\n            outputFile.write(\"{}\\n\".format(json.dumps(outPut, sort_keys=True)))\n            outputFile.flush()\n    except Exception:\n        traceback.print_exc()\n    finally:\n        os.umask(old_umask)\n\n\nif __name__ == \"__main__\":\n    main(sys.argv[1:])\n", "path": "tools/nni_gpu_tool/gpu_metrics_collector.py"}]}