We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[tune] AssertionError: Resource invalid
<!--
General questions should be asked on the mailing list ray-dev@googlegroups.com.
Questions about how to use Ray should be asked on
[StackOverflow](https://stackoverflow.com/questions/tagged/ray).

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
Ubuntu 16.04
- **Ray installed from (source or binary)**:
pip install https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.8.0.dev4-cp36-cp36m-manylinux1_x86_64.whl
- **Ray version**:
0.8.0.dev4
- **Python version**:
3.6.7
- **Exact command to reproduce**:

<!--
You can obtain the Ray version with

python -c "import ray; print(ray.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

I run 5 trials with ray.tune. In one of the trials (each time), an error occurs at the end of training: `AssertionError: Resource invalid: Resources(cpu=3, gpu=0.33, memory=0, object_store_memory=0, extra_cpu=0, extra_gpu=0, extra_memory=0, extra_object_store_memory=0, custom_resources={}, extra_custom_resources={})`.

When I trace back the error, I end up in the following function (ray/tune/resources.py):

```
def is_nonnegative(self):
    all_values = [self.cpu, self.gpu, self.extra_cpu, self.extra_gpu]
    all_values += list(self.custom_resources.values())
    all_values += list(self.extra_custom_resources.values())
    return all(v >= 0 for v in all_values)
```

It seems `custom_resources` and `extra_custom_resources` are not defined. It is weird that the error only occurs in one run... Is this a bug, or any suggestions on how to fix?


### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->

__This is how I call `tune.run`__

```
tune.run(
    ModelTrainerMT,
    resources_per_trial={
        'cpu': config['ncpu'],
        'gpu': config['ngpu'],
    },
    num_samples=1,
    config=best_config,
    local_dir=store,
    raise_on_failed_trial=True,
    verbose=1,
    with_server=False,
    ray_auto_init=False,
    scheduler=early_stopping_scheduler,
    loggers=[JsonLogger, CSVLogger],
    checkpoint_at_end=True,
    reuse_actors=True,
    stop={'epoch': 2 if args.test else config['max_t']}
)
```

__Traceback__

```
2019-09-06 09:56:45,526 ERROR trial_runner.py:557 -- Error processing event.
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 552, in _process_trial
    self.trial_executor.stop_trial(trial)
  File "/opt/conda/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 246, in stop_trial
    self._return_resources(trial.resources)
  File "/opt/conda/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 388, in _return_resources
    "Resource invalid: {}".format(resources))
AssertionError: Resource invalid: Resources(cpu=3, gpu=0.33, memory=0, object_store_memory=0, extra_cpu=0, extra_gpu=0, extra_memory=0, extra_object_store_memory=0, custom_resources={}, extra_custom_resources={})
```




--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `python/ray/tune/resources.py`
Content:
```
1 from __future__ import absolute_import
2 from __future__ import division
3 from __future__ import print_function
4 
5 from collections import namedtuple
6 import logging
7 import json
8 # For compatibility under py2 to consider unicode as str
9 from six import string_types
10 
11 from numbers import Number
12 
13 from ray.tune import TuneError
14 
15 logger = logging.getLogger(__name__)
16 
17 
18 class Resources(
19         namedtuple("Resources", [
20             "cpu", "gpu", "memory", "object_store_memory", "extra_cpu",
21             "extra_gpu", "extra_memory", "extra_object_store_memory",
22             "custom_resources", "extra_custom_resources"
23         ])):
24     """Ray resources required to schedule a trial.
25 
26     Attributes:
27         cpu (float): Number of CPUs to allocate to the trial.
28         gpu (float): Number of GPUs to allocate to the trial.
29         memory (float): Memory to reserve for the trial.
30         object_store_memory (float): Object store memory to reserve.
31         extra_cpu (float): Extra CPUs to reserve in case the trial needs to
32             launch additional Ray actors that use CPUs.
33         extra_gpu (float): Extra GPUs to reserve in case the trial needs to
34             launch additional Ray actors that use GPUs.
35         extra_memory (float): Memory to reserve for the trial launching
36             additional Ray actors that use memory.
37         extra_object_store_memory (float): Object store memory to reserve for
38             the trial launching additional Ray actors that use object store
39             memory.
40         custom_resources (dict): Mapping of resource to quantity to allocate
41             to the trial.
42         extra_custom_resources (dict): Extra custom resources to reserve in
43             case the trial needs to launch additional Ray actors that use
44             any of these custom resources.
45 
46     """
47 
48     __slots__ = ()
49 
50     def __new__(cls,
51                 cpu,
52                 gpu,
53                 memory=0,
54                 object_store_memory=0,
55                 extra_cpu=0,
56                 extra_gpu=0,
57                 extra_memory=0,
58                 extra_object_store_memory=0,
59                 custom_resources=None,
60                 extra_custom_resources=None):
61         custom_resources = custom_resources or {}
62         extra_custom_resources = extra_custom_resources or {}
63         leftovers = set(custom_resources) ^ set(extra_custom_resources)
64 
65         for value in leftovers:
66             custom_resources.setdefault(value, 0)
67             extra_custom_resources.setdefault(value, 0)
68 
69         all_values = [
70             cpu, gpu, memory, object_store_memory, extra_cpu, extra_gpu,
71             extra_memory, extra_object_store_memory
72         ]
73         all_values += list(custom_resources.values())
74         all_values += list(extra_custom_resources.values())
75         assert len(custom_resources) == len(extra_custom_resources)
76         for entry in all_values:
77             assert isinstance(entry, Number), ("Improper resource value.",
78                                                entry)
79         return super(Resources, cls).__new__(
80             cls, cpu, gpu, memory, object_store_memory, extra_cpu, extra_gpu,
81             extra_memory, extra_object_store_memory, custom_resources,
82             extra_custom_resources)
83 
84     def summary_string(self):
85         summary = "{} CPUs, {} GPUs".format(self.cpu + self.extra_cpu,
86                                             self.gpu + self.extra_gpu)
87         if self.memory or self.extra_memory:
88             summary += ", {} GiB heap".format(
89                 round((self.memory + self.extra_memory) / (1024**3), 2))
90         if self.object_store_memory or self.extra_object_store_memory:
91             summary += ", {} GiB objects".format(
92                 round(
93                     (self.object_store_memory + self.extra_object_store_memory)
94                     / (1024**3), 2))
95         custom_summary = ", ".join([
96             "{} {}".format(self.get_res_total(res), res)
97             for res in self.custom_resources
98         ])
99         if custom_summary:
100             summary += " ({})".format(custom_summary)
101         return summary
102 
103     def cpu_total(self):
104         return self.cpu + self.extra_cpu
105 
106     def gpu_total(self):
107         return self.gpu + self.extra_gpu
108 
109     def memory_total(self):
110         return self.memory + self.extra_memory
111 
112     def object_store_memory_total(self):
113         return self.object_store_memory + self.extra_object_store_memory
114 
115     def get_res_total(self, key):
116         return self.custom_resources.get(
117             key, 0) + self.extra_custom_resources.get(key, 0)
118 
119     def get(self, key):
120         return self.custom_resources.get(key, 0)
121 
122     def is_nonnegative(self):
123         all_values = [self.cpu, self.gpu, self.extra_cpu, self.extra_gpu]
124         all_values += list(self.custom_resources.values())
125         all_values += list(self.extra_custom_resources.values())
126         return all(v >= 0 for v in all_values)
127 
128     @classmethod
129     def subtract(cls, original, to_remove):
130         cpu = original.cpu - to_remove.cpu
131         gpu = original.gpu - to_remove.gpu
132         memory = original.memory - to_remove.memory
133         object_store_memory = (
134             original.object_store_memory - to_remove.object_store_memory)
135         extra_cpu = original.extra_cpu - to_remove.extra_cpu
136         extra_gpu = original.extra_gpu - to_remove.extra_gpu
137         extra_memory = original.extra_memory - to_remove.extra_memory
138         extra_object_store_memory = (original.extra_object_store_memory -
139                                      to_remove.extra_object_store_memory)
140         all_resources = set(original.custom_resources).union(
141             set(to_remove.custom_resources))
142         new_custom_res = {
143             k: original.custom_resources.get(k, 0) -
144             to_remove.custom_resources.get(k, 0)
145             for k in all_resources
146         }
147         extra_custom_res = {
148             k: original.extra_custom_resources.get(k, 0) -
149             to_remove.extra_custom_resources.get(k, 0)
150             for k in all_resources
151         }
152         return Resources(cpu, gpu, memory, object_store_memory, extra_cpu,
153                          extra_gpu, extra_memory, extra_object_store_memory,
154                          new_custom_res, extra_custom_res)
155 
156     def to_json(self):
157         return resources_to_json(self)
158 
159 
160 def json_to_resources(data):
161     if data is None or data == "null":
162         return None
163     if isinstance(data, string_types):
164         data = json.loads(data)
165     for k in data:
166         if k in ["driver_cpu_limit", "driver_gpu_limit"]:
167             raise TuneError(
168                 "The field `{}` is no longer supported. Use `extra_cpu` "
169                 "or `extra_gpu` instead.".format(k))
170         if k not in Resources._fields:
171             raise ValueError(
172                 "Unknown resource field {}, must be one of {}".format(
173                     k, Resources._fields))
174     return Resources(
175         data.get("cpu", 1), data.get("gpu", 0), data.get("memory", 0),
176         data.get("object_store_memory", 0), data.get("extra_cpu", 0),
177         data.get("extra_gpu", 0), data.get("extra_memory", 0),
178         data.get("extra_object_store_memory", 0), data.get("custom_resources"),
179         data.get("extra_custom_resources"))
180 
181 
182 def resources_to_json(resources):
183     if resources is None:
184         return None
185     return {
186         "cpu": resources.cpu,
187         "gpu": resources.gpu,
188         "memory": resources.memory,
189         "object_store_memory": resources.object_store_memory,
190         "extra_cpu": resources.extra_cpu,
191         "extra_gpu": resources.extra_gpu,
192         "extra_memory": resources.extra_memory,
193         "extra_object_store_memory": resources.extra_object_store_memory,
194         "custom_resources": resources.custom_resources.copy(),
195         "extra_custom_resources": resources.extra_custom_resources.copy()
196     }
197
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/python/ray/tune/resources.py b/python/ray/tune/resources.py
--- a/python/ray/tune/resources.py
+++ b/python/ray/tune/resources.py
@@ -5,11 +5,10 @@
 from collections import namedtuple
 import logging
 import json
+from numbers import Number
 # For compatibility under py2 to consider unicode as str
 from six import string_types
 
-from numbers import Number
-
 from ray.tune import TuneError
 
 logger = logging.getLogger(__name__)
@@ -66,6 +65,23 @@
             custom_resources.setdefault(value, 0)
             extra_custom_resources.setdefault(value, 0)
 
+        cpu = round(cpu, 2)
+        gpu = round(gpu, 2)
+        memory = round(memory, 2)
+        object_store_memory = round(object_store_memory, 2)
+        extra_cpu = round(extra_cpu, 2)
+        extra_gpu = round(extra_gpu, 2)
+        extra_memory = round(extra_memory, 2)
+        extra_object_store_memory = round(extra_object_store_memory, 2)
+        custom_resources = {
+            resource: round(value, 2)
+            for resource, value in custom_resources.items()
+        }
+        extra_custom_resources = {
+            resource: round(value, 2)
+            for resource, value in extra_custom_resources.items()
+        }
+
         all_values = [
             cpu, gpu, memory, object_store_memory, extra_cpu, extra_gpu,
             extra_memory, extra_object_store_memory

{"golden_diff": "diff --git a/python/ray/tune/resources.py b/python/ray/tune/resources.py\n--- a/python/ray/tune/resources.py\n+++ b/python/ray/tune/resources.py\n@@ -5,11 +5,10 @@\n from collections import namedtuple\n import logging\n import json\n+from numbers import Number\n # For compatibility under py2 to consider unicode as str\n from six import string_types\n \n-from numbers import Number\n-\n from ray.tune import TuneError\n \n logger = logging.getLogger(__name__)\n@@ -66,6 +65,23 @@\n             custom_resources.setdefault(value, 0)\n             extra_custom_resources.setdefault(value, 0)\n \n+        cpu = round(cpu, 2)\n+        gpu = round(gpu, 2)\n+        memory = round(memory, 2)\n+        object_store_memory = round(object_store_memory, 2)\n+        extra_cpu = round(extra_cpu, 2)\n+        extra_gpu = round(extra_gpu, 2)\n+        extra_memory = round(extra_memory, 2)\n+        extra_object_store_memory = round(extra_object_store_memory, 2)\n+        custom_resources = {\n+            resource: round(value, 2)\n+            for resource, value in custom_resources.items()\n+        }\n+        extra_custom_resources = {\n+            resource: round(value, 2)\n+            for resource, value in extra_custom_resources.items()\n+        }\n+\n         all_values = [\n             cpu, gpu, memory, object_store_memory, extra_cpu, extra_gpu,\n             extra_memory, extra_object_store_memory\n", "issue": "[tune] AssertionError: Resource invalid\n<!--\r\nGeneral questions should be asked on the mailing list ray-dev@googlegroups.com.\r\nQuestions about how to use Ray should be asked on\r\n[StackOverflow](https://stackoverflow.com/questions/tagged/ray).\r\n\r\nBefore submitting an issue, please fill out the following form.\r\n-->\r\n\r\n### System information\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:\r\nUbuntu 16.04\r\n- **Ray installed from (source or binary)**:\r\npip install https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.8.0.dev4-cp36-cp36m-manylinux1_x86_64.whl\r\n- **Ray version**:\r\n0.8.0.dev4\r\n- **Python version**:\r\n3.6.7\r\n- **Exact command to reproduce**:\r\n\r\n<!--\r\nYou can obtain the Ray version with\r\n\r\npython -c \"import ray; print(ray.__version__)\"\r\n-->\r\n\r\n### Describe the problem\r\n<!-- Describe the problem clearly here. -->\r\n\r\nI run 5 trials with ray.tune. In one of the trials (each time), an error occurs at the end of training: `AssertionError: Resource invalid: Resources(cpu=3, gpu=0.33, memory=0, object_store_memory=0, extra_cpu=0, extra_gpu=0, extra_memory=0, extra_object_store_memory=0, custom_resources={}, extra_custom_resources={})`.\r\n\r\nWhen I trace back the error, I end up in the following function (ray/tune/resources.py):\r\n\r\n```\r\ndef is_nonnegative(self):\r\n    all_values = [self.cpu, self.gpu, self.extra_cpu, self.extra_gpu]\r\n    all_values += list(self.custom_resources.values())\r\n    all_values += list(self.extra_custom_resources.values())\r\n    return all(v >= 0 for v in all_values)\r\n```\r\n\r\nIt seems `custom_resources` and `extra_custom_resources` are not defined. It is weird that the error only occurs in one run... Is this a bug, or any suggestions on how to fix?\r\n\r\n\r\n### Source code / logs\r\n<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->\r\n\r\n__This is how I call `tune.run`__\r\n\r\n```\r\ntune.run(\r\n    ModelTrainerMT,\r\n    resources_per_trial={\r\n        'cpu': config['ncpu'],\r\n        'gpu': config['ngpu'],\r\n    },\r\n    num_samples=1,\r\n    config=best_config,\r\n    local_dir=store,\r\n    raise_on_failed_trial=True,\r\n    verbose=1,\r\n    with_server=False,\r\n    ray_auto_init=False,\r\n    scheduler=early_stopping_scheduler,\r\n    loggers=[JsonLogger, CSVLogger],\r\n    checkpoint_at_end=True,\r\n    reuse_actors=True,\r\n    stop={'epoch': 2 if args.test else config['max_t']}\r\n)\r\n```\r\n\r\n__Traceback__\r\n\r\n```\r\n2019-09-06 09:56:45,526 ERROR trial_runner.py:557 -- Error processing event.\r\nTraceback (most recent call last):\r\n  File \"/opt/conda/lib/python3.6/site-packages/ray/tune/trial_runner.py\", line 552, in _process_trial\r\n    self.trial_executor.stop_trial(trial)\r\n  File \"/opt/conda/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py\", line 246, in stop_trial\r\n    self._return_resources(trial.resources)\r\n  File \"/opt/conda/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py\", line 388, in _return_resources\r\n    \"Resource invalid: {}\".format(resources))\r\nAssertionError: Resource invalid: Resources(cpu=3, gpu=0.33, memory=0, object_store_memory=0, extra_cpu=0, extra_gpu=0, extra_memory=0, extra_object_store_memory=0, custom_resources={}, extra_custom_resources={})\r\n```\r\n\r\n\r\n\n", "before_files": [{"content": "from __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nfrom collections import namedtuple\nimport logging\nimport json\n# For compatibility under py2 to consider unicode as str\nfrom six import string_types\n\nfrom numbers import Number\n\nfrom ray.tune import TuneError\n\nlogger = logging.getLogger(__name__)\n\n\nclass Resources(\n        namedtuple(\"Resources\", [\n            \"cpu\", \"gpu\", \"memory\", \"object_store_memory\", \"extra_cpu\",\n            \"extra_gpu\", \"extra_memory\", \"extra_object_store_memory\",\n            \"custom_resources\", \"extra_custom_resources\"\n        ])):\n    \"\"\"Ray resources required to schedule a trial.\n\n    Attributes:\n        cpu (float): Number of CPUs to allocate to the trial.\n        gpu (float): Number of GPUs to allocate to the trial.\n        memory (float): Memory to reserve for the trial.\n        object_store_memory (float): Object store memory to reserve.\n        extra_cpu (float): Extra CPUs to reserve in case the trial needs to\n            launch additional Ray actors that use CPUs.\n        extra_gpu (float): Extra GPUs to reserve in case the trial needs to\n            launch additional Ray actors that use GPUs.\n        extra_memory (float): Memory to reserve for the trial launching\n            additional Ray actors that use memory.\n        extra_object_store_memory (float): Object store memory to reserve for\n            the trial launching additional Ray actors that use object store\n            memory.\n        custom_resources (dict): Mapping of resource to quantity to allocate\n            to the trial.\n        extra_custom_resources (dict): Extra custom resources to reserve in\n            case the trial needs to launch additional Ray actors that use\n            any of these custom resources.\n\n    \"\"\"\n\n    __slots__ = ()\n\n    def __new__(cls,\n                cpu,\n                gpu,\n                memory=0,\n                object_store_memory=0,\n                extra_cpu=0,\n                extra_gpu=0,\n                extra_memory=0,\n                extra_object_store_memory=0,\n                custom_resources=None,\n                extra_custom_resources=None):\n        custom_resources = custom_resources or {}\n        extra_custom_resources = extra_custom_resources or {}\n        leftovers = set(custom_resources) ^ set(extra_custom_resources)\n\n        for value in leftovers:\n            custom_resources.setdefault(value, 0)\n            extra_custom_resources.setdefault(value, 0)\n\n        all_values = [\n            cpu, gpu, memory, object_store_memory, extra_cpu, extra_gpu,\n            extra_memory, extra_object_store_memory\n        ]\n        all_values += list(custom_resources.values())\n        all_values += list(extra_custom_resources.values())\n        assert len(custom_resources) == len(extra_custom_resources)\n        for entry in all_values:\n            assert isinstance(entry, Number), (\"Improper resource value.\",\n                                               entry)\n        return super(Resources, cls).__new__(\n            cls, cpu, gpu, memory, object_store_memory, extra_cpu, extra_gpu,\n            extra_memory, extra_object_store_memory, custom_resources,\n            extra_custom_resources)\n\n    def summary_string(self):\n        summary = \"{} CPUs, {} GPUs\".format(self.cpu + self.extra_cpu,\n                                            self.gpu + self.extra_gpu)\n        if self.memory or self.extra_memory:\n            summary += \", {} GiB heap\".format(\n                round((self.memory + self.extra_memory) / (1024**3), 2))\n        if self.object_store_memory or self.extra_object_store_memory:\n            summary += \", {} GiB objects\".format(\n                round(\n                    (self.object_store_memory + self.extra_object_store_memory)\n                    / (1024**3), 2))\n        custom_summary = \", \".join([\n            \"{} {}\".format(self.get_res_total(res), res)\n            for res in self.custom_resources\n        ])\n        if custom_summary:\n            summary += \" ({})\".format(custom_summary)\n        return summary\n\n    def cpu_total(self):\n        return self.cpu + self.extra_cpu\n\n    def gpu_total(self):\n        return self.gpu + self.extra_gpu\n\n    def memory_total(self):\n        return self.memory + self.extra_memory\n\n    def object_store_memory_total(self):\n        return self.object_store_memory + self.extra_object_store_memory\n\n    def get_res_total(self, key):\n        return self.custom_resources.get(\n            key, 0) + self.extra_custom_resources.get(key, 0)\n\n    def get(self, key):\n        return self.custom_resources.get(key, 0)\n\n    def is_nonnegative(self):\n        all_values = [self.cpu, self.gpu, self.extra_cpu, self.extra_gpu]\n        all_values += list(self.custom_resources.values())\n        all_values += list(self.extra_custom_resources.values())\n        return all(v >= 0 for v in all_values)\n\n    @classmethod\n    def subtract(cls, original, to_remove):\n        cpu = original.cpu - to_remove.cpu\n        gpu = original.gpu - to_remove.gpu\n        memory = original.memory - to_remove.memory\n        object_store_memory = (\n            original.object_store_memory - to_remove.object_store_memory)\n        extra_cpu = original.extra_cpu - to_remove.extra_cpu\n        extra_gpu = original.extra_gpu - to_remove.extra_gpu\n        extra_memory = original.extra_memory - to_remove.extra_memory\n        extra_object_store_memory = (original.extra_object_store_memory -\n                                     to_remove.extra_object_store_memory)\n        all_resources = set(original.custom_resources).union(\n            set(to_remove.custom_resources))\n        new_custom_res = {\n            k: original.custom_resources.get(k, 0) -\n            to_remove.custom_resources.get(k, 0)\n            for k in all_resources\n        }\n        extra_custom_res = {\n            k: original.extra_custom_resources.get(k, 0) -\n            to_remove.extra_custom_resources.get(k, 0)\n            for k in all_resources\n        }\n        return Resources(cpu, gpu, memory, object_store_memory, extra_cpu,\n                         extra_gpu, extra_memory, extra_object_store_memory,\n                         new_custom_res, extra_custom_res)\n\n    def to_json(self):\n        return resources_to_json(self)\n\n\ndef json_to_resources(data):\n    if data is None or data == \"null\":\n        return None\n    if isinstance(data, string_types):\n        data = json.loads(data)\n    for k in data:\n        if k in [\"driver_cpu_limit\", \"driver_gpu_limit\"]:\n            raise TuneError(\n                \"The field `{}` is no longer supported. Use `extra_cpu` \"\n                \"or `extra_gpu` instead.\".format(k))\n        if k not in Resources._fields:\n            raise ValueError(\n                \"Unknown resource field {}, must be one of {}\".format(\n                    k, Resources._fields))\n    return Resources(\n        data.get(\"cpu\", 1), data.get(\"gpu\", 0), data.get(\"memory\", 0),\n        data.get(\"object_store_memory\", 0), data.get(\"extra_cpu\", 0),\n        data.get(\"extra_gpu\", 0), data.get(\"extra_memory\", 0),\n        data.get(\"extra_object_store_memory\", 0), data.get(\"custom_resources\"),\n        data.get(\"extra_custom_resources\"))\n\n\ndef resources_to_json(resources):\n    if resources is None:\n        return None\n    return {\n        \"cpu\": resources.cpu,\n        \"gpu\": resources.gpu,\n        \"memory\": resources.memory,\n        \"object_store_memory\": resources.object_store_memory,\n        \"extra_cpu\": resources.extra_cpu,\n        \"extra_gpu\": resources.extra_gpu,\n        \"extra_memory\": resources.extra_memory,\n        \"extra_object_store_memory\": resources.extra_object_store_memory,\n        \"custom_resources\": resources.custom_resources.copy(),\n        \"extra_custom_resources\": resources.extra_custom_resources.copy()\n    }\n", "path": "python/ray/tune/resources.py"}], "after_files": [{"content": "from __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nfrom collections import namedtuple\nimport logging\nimport json\nfrom numbers import Number\n# For compatibility under py2 to consider unicode as str\nfrom six import string_types\n\nfrom ray.tune import TuneError\n\nlogger = logging.getLogger(__name__)\n\n\nclass Resources(\n        namedtuple(\"Resources\", [\n            \"cpu\", \"gpu\", \"memory\", \"object_store_memory\", \"extra_cpu\",\n            \"extra_gpu\", \"extra_memory\", \"extra_object_store_memory\",\n            \"custom_resources\", \"extra_custom_resources\"\n        ])):\n    \"\"\"Ray resources required to schedule a trial.\n\n    Attributes:\n        cpu (float): Number of CPUs to allocate to the trial.\n        gpu (float): Number of GPUs to allocate to the trial.\n        memory (float): Memory to reserve for the trial.\n        object_store_memory (float): Object store memory to reserve.\n        extra_cpu (float): Extra CPUs to reserve in case the trial needs to\n            launch additional Ray actors that use CPUs.\n        extra_gpu (float): Extra GPUs to reserve in case the trial needs to\n            launch additional Ray actors that use GPUs.\n        extra_memory (float): Memory to reserve for the trial launching\n            additional Ray actors that use memory.\n        extra_object_store_memory (float): Object store memory to reserve for\n            the trial launching additional Ray actors that use object store\n            memory.\n        custom_resources (dict): Mapping of resource to quantity to allocate\n            to the trial.\n        extra_custom_resources (dict): Extra custom resources to reserve in\n            case the trial needs to launch additional Ray actors that use\n            any of these custom resources.\n\n    \"\"\"\n\n    __slots__ = ()\n\n    def __new__(cls,\n                cpu,\n                gpu,\n                memory=0,\n                object_store_memory=0,\n                extra_cpu=0,\n                extra_gpu=0,\n                extra_memory=0,\n                extra_object_store_memory=0,\n                custom_resources=None,\n                extra_custom_resources=None):\n        custom_resources = custom_resources or {}\n        extra_custom_resources = extra_custom_resources or {}\n        leftovers = set(custom_resources) ^ set(extra_custom_resources)\n\n        for value in leftovers:\n            custom_resources.setdefault(value, 0)\n            extra_custom_resources.setdefault(value, 0)\n\n        cpu = round(cpu, 2)\n        gpu = round(gpu, 2)\n        memory = round(memory, 2)\n        object_store_memory = round(object_store_memory, 2)\n        extra_cpu = round(extra_cpu, 2)\n        extra_gpu = round(extra_gpu, 2)\n        extra_memory = round(extra_memory, 2)\n        extra_object_store_memory = round(extra_object_store_memory, 2)\n        custom_resources = {\n            resource: round(value, 2)\n            for resource, value in custom_resources.items()\n        }\n        extra_custom_resources = {\n            resource: round(value, 2)\n            for resource, value in extra_custom_resources.items()\n        }\n\n        all_values = [\n            cpu, gpu, memory, object_store_memory, extra_cpu, extra_gpu,\n            extra_memory, extra_object_store_memory\n        ]\n        all_values += list(custom_resources.values())\n        all_values += list(extra_custom_resources.values())\n        assert len(custom_resources) == len(extra_custom_resources)\n        for entry in all_values:\n            assert isinstance(entry, Number), (\"Improper resource value.\",\n                                               entry)\n        return super(Resources, cls).__new__(\n            cls, cpu, gpu, memory, object_store_memory, extra_cpu, extra_gpu,\n            extra_memory, extra_object_store_memory, custom_resources,\n            extra_custom_resources)\n\n    def summary_string(self):\n        summary = \"{} CPUs, {} GPUs\".format(self.cpu + self.extra_cpu,\n                                            self.gpu + self.extra_gpu)\n        if self.memory or self.extra_memory:\n            summary += \", {} GiB heap\".format(\n                round((self.memory + self.extra_memory) / (1024**3), 2))\n        if self.object_store_memory or self.extra_object_store_memory:\n            summary += \", {} GiB objects\".format(\n                round(\n                    (self.object_store_memory + self.extra_object_store_memory)\n                    / (1024**3), 2))\n        custom_summary = \", \".join([\n            \"{} {}\".format(self.get_res_total(res), res)\n            for res in self.custom_resources\n        ])\n        if custom_summary:\n            summary += \" ({})\".format(custom_summary)\n        return summary\n\n    def cpu_total(self):\n        return self.cpu + self.extra_cpu\n\n    def gpu_total(self):\n        return self.gpu + self.extra_gpu\n\n    def memory_total(self):\n        return self.memory + self.extra_memory\n\n    def object_store_memory_total(self):\n        return self.object_store_memory + self.extra_object_store_memory\n\n    def get_res_total(self, key):\n        return self.custom_resources.get(\n            key, 0) + self.extra_custom_resources.get(key, 0)\n\n    def get(self, key):\n        return self.custom_resources.get(key, 0)\n\n    def is_nonnegative(self):\n        all_values = [self.cpu, self.gpu, self.extra_cpu, self.extra_gpu]\n        all_values += list(self.custom_resources.values())\n        all_values += list(self.extra_custom_resources.values())\n        return all(v >= 0 for v in all_values)\n\n    @classmethod\n    def subtract(cls, original, to_remove):\n        cpu = original.cpu - to_remove.cpu\n        gpu = original.gpu - to_remove.gpu\n        memory = original.memory - to_remove.memory\n        object_store_memory = (\n            original.object_store_memory - to_remove.object_store_memory)\n        extra_cpu = original.extra_cpu - to_remove.extra_cpu\n        extra_gpu = original.extra_gpu - to_remove.extra_gpu\n        extra_memory = original.extra_memory - to_remove.extra_memory\n        extra_object_store_memory = (original.extra_object_store_memory -\n                                     to_remove.extra_object_store_memory)\n        all_resources = set(original.custom_resources).union(\n            set(to_remove.custom_resources))\n        new_custom_res = {\n            k: original.custom_resources.get(k, 0) -\n            to_remove.custom_resources.get(k, 0)\n            for k in all_resources\n        }\n        extra_custom_res = {\n            k: original.extra_custom_resources.get(k, 0) -\n            to_remove.extra_custom_resources.get(k, 0)\n            for k in all_resources\n        }\n        return Resources(cpu, gpu, memory, object_store_memory, extra_cpu,\n                         extra_gpu, extra_memory, extra_object_store_memory,\n                         new_custom_res, extra_custom_res)\n\n    def to_json(self):\n        return resources_to_json(self)\n\n\ndef json_to_resources(data):\n    if data is None or data == \"null\":\n        return None\n    if isinstance(data, string_types):\n        data = json.loads(data)\n    for k in data:\n        if k in [\"driver_cpu_limit\", \"driver_gpu_limit\"]:\n            raise TuneError(\n                \"The field `{}` is no longer supported. Use `extra_cpu` \"\n                \"or `extra_gpu` instead.\".format(k))\n        if k not in Resources._fields:\n            raise ValueError(\n                \"Unknown resource field {}, must be one of {}\".format(\n                    k, Resources._fields))\n    return Resources(\n        data.get(\"cpu\", 1), data.get(\"gpu\", 0), data.get(\"memory\", 0),\n        data.get(\"object_store_memory\", 0), data.get(\"extra_cpu\", 0),\n        data.get(\"extra_gpu\", 0), data.get(\"extra_memory\", 0),\n        data.get(\"extra_object_store_memory\", 0), data.get(\"custom_resources\"),\n        data.get(\"extra_custom_resources\"))\n\n\ndef resources_to_json(resources):\n    if resources is None:\n        return None\n    return {\n        \"cpu\": resources.cpu,\n        \"gpu\": resources.gpu,\n        \"memory\": resources.memory,\n        \"object_store_memory\": resources.object_store_memory,\n        \"extra_cpu\": resources.extra_cpu,\n        \"extra_gpu\": resources.extra_gpu,\n        \"extra_memory\": resources.extra_memory,\n        \"extra_object_store_memory\": resources.extra_object_store_memory,\n        \"custom_resources\": resources.custom_resources.copy(),\n        \"extra_custom_resources\": resources.extra_custom_resources.copy()\n    }\n", "path": "python/ray/tune/resources.py"}]}