We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Add custom user-agent to `NetworkClient` and external API calls
<!--
Instructions:
 * Fill out the sections below, replace …'s with information about your issue
 * Use the 'preview' function above this text box to verify formatting before submitting
-->

## Observed behavior
<!--
Description of the behavior that was observed, including screenshots or other references when applicable
-->
The user agent for API calls made with python `requests` do not clearly identify the device as Kolibri

## Errors and logs
<!--
Relevant logs from:
 * the command line
 * ~/.kolibri/logs/kolibri.txt
 * the browser console

Please wrap errors in triple backticks for clean formatting like this:
```
01:10 info: something happened
01:12 error: something bad happened
```
-->
Perhaps connected with errors like the following although these are not the primary reason to customize the user-agent:
```
http.client.RemoteDisconnected: Remote end closed connection without response
```

## Expected behavior
<!--
Description of what behavior was expected but did not occur
-->
It's recommended that we clearly identify the application making API calls, to differentiate it from potential scripting. According to MDN, it should have the format:
```
User-Agent: <product> / <product-version> <comment>
```
So for Kolibri 0.16.0:
```
User-Agent: kolibri/0.16.0 python-requests/2.28.2
```

## User-facing consequences
<!--
Implications and real-world consequences for learners, coaches, admins, and other users of the application
-->
Web application firewalls may take more aggressive action against clients making many requests if it appears to be purely something scripted.


## Context
<!--
Tell us about your environment, including:
 * Kolibri version
 * Operating system
 * Browser
-->
Kolibri 0.15+


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `kolibri/core/discovery/utils/network/client.py`
Content:
```
1 import logging
2 
3 import requests
4 from six import raise_from
5 from six.moves.urllib.parse import urlparse
6 
7 from . import errors
8 from .urls import get_normalized_url_variations
9 from .urls import HTTP_PORTS
10 from .urls import HTTPS_PORTS
11 from kolibri.core.discovery.models import ConnectionStatus
12 from kolibri.core.tasks.utils import get_current_job
13 from kolibri.core.utils.urls import join_url
14 from kolibri.utils.server import get_urls
15 
16 logger = logging.getLogger(__name__)
17 
18 device_info_defaults = {
19     "subset_of_users_device": False,
20 }
21 
22 DEFAULT_CONNECT_TIMEOUT = 5
23 DEFAULT_READ_TIMEOUT = 60
24 # default read timeout when within a job
25 DEFAULT_ASYNC_READ_TIMEOUT = 30
26 # when the network client tries variations of a url, that means the overall length of time it takes
27 # is multiplied by the number of variations, so for synchronous operations (in a HTTP request) we
28 # make the overall timeout ~= the DEFAULT_READ_TIMEOUT
29 DEFAULT_SYNC_READ_TIMEOUT = DEFAULT_READ_TIMEOUT / (len(HTTP_PORTS) + len(HTTPS_PORTS))
30 
31 
32 class NetworkClient(requests.Session):
33     __slots__ = ("base_url", "timeout", "session", "device_info", "remote_ip")
34 
35     def __init__(self, base_url, timeout=None):
36         """
37         If an explicit base_url is already known, provide that. If only a vague address is known,
38         `build_from_address` can build a client to determine the actual `base_url`
39         :param base_url: The fully composed URL for a network location, without path
40         :param timeout: A timeout value in seconds or tuple for (connect, read)
41         :type timeout: float|tuple
42         """
43         super(NetworkClient, self).__init__()
44 
45         self.base_url = base_url
46         self.timeout = timeout or (DEFAULT_CONNECT_TIMEOUT, DEFAULT_READ_TIMEOUT)
47         self.session = None
48         self.device_info = None
49         self.remote_ip = None
50 
51     @classmethod
52     def build_for_address(cls, address, timeout=None):
53         """
54         Normalizes the address URL and tries a number of variations until we find one
55         that's able to connect
56 
57         :param address: The address of which to try variations of
58         :param timeout: A timeout value in seconds or tuple for (connect, read)
59         :return: A NetworkClient with a verified connection
60         :rtype: NetworkClient
61         """
62         logger.info(
63             "Attempting connections to variations of the URL: {}".format(address)
64         )
65         if timeout is None:
66             if get_current_job() is not None:
67                 # when we're within a job, then we can use longer timeouts
68                 timeout = (DEFAULT_CONNECT_TIMEOUT, DEFAULT_ASYNC_READ_TIMEOUT)
69             else:
70                 # if we're within a request thread, then we limit it for an overall time
71                 timeout = (DEFAULT_CONNECT_TIMEOUT, DEFAULT_SYNC_READ_TIMEOUT)
72         _, self_urls = get_urls()
73         for url in get_normalized_url_variations(address):
74             if url in self_urls:
75                 continue  # exclude our own URLs
76             with NetworkClient(url, timeout=timeout) as client:
77                 if client.connect(raise_if_unavailable=False):
78                     return client
79         # we weren't able to connect to any of the URL variations, so all we can do is throw
80         raise errors.NetworkLocationNotFound()
81 
82     @classmethod
83     def build_from_network_location(cls, network_location, timeout=None):
84         """
85         Creates a NetworkClient for a NetworkLocation, and validates the connection if the status
86         isn't already 'Okay'
87         :param network_location: The network location model
88         :type network_location: kolibri.core.discovery.models.NetworkLocation
89         :param timeout: A timeout value in seconds or tuple for (connect, read)
90         :return: A NetworkClient with a verified connection
91         :rtype: NetworkClient
92         """
93         # expect that static network locations have an exact base_url, and only try different
94         # variations if we haven't already
95         if (
96             network_location.dynamic
97             and network_location.connection_status == ConnectionStatus.Unknown
98         ):
99             return NetworkClient.build_for_address(
100                 network_location.base_url, timeout=timeout
101             )
102         return NetworkClient(network_location.base_url, timeout=timeout)
103 
104     def head(self, path, **kwargs):
105         return self.request("HEAD", path, **kwargs)
106 
107     def get(self, path, **kwargs):
108         return self.request("GET", path, **kwargs)
109 
110     def post(self, path, **kwargs):
111         return self.request("POST", path, **kwargs)
112 
113     def request(self, method, path, **kwargs):
114         response = None
115         if "timeout" not in kwargs:
116             kwargs.update(timeout=self.timeout)
117 
118         url = join_url(self.base_url, path)
119         try:
120             with super(NetworkClient, self).request(
121                 method, url, stream=True, **kwargs
122             ) as response:
123                 if response.raw._connection.sock is None:
124                     raise requests.exceptions.ConnectionError("No socket available")
125 
126                 # capture the remote IP address, which requires `stream=True` and before consumed
127                 self.remote_ip = response.raw._connection.sock.getpeername()[0]
128                 # now consume content, see how `Session.send` does this when `stream=False`
129                 response.content
130 
131             response.raise_for_status()
132             return response
133         except (
134             requests.exceptions.ConnectionError,
135             requests.exceptions.SSLError,
136             requests.exceptions.ConnectTimeout,
137             requests.exceptions.URLRequired,
138             requests.exceptions.MissingSchema,
139             requests.exceptions.InvalidSchema,
140             requests.exceptions.InvalidURL,
141             requests.exceptions.InvalidHeader,
142             requests.exceptions.InvalidJSONError,
143         ) as e:
144             raise_from(
145                 errors.NetworkLocationConnectionFailure(
146                     "Unable to connect: {}".format(url)
147                 ),
148                 e,
149             )
150         except (
151             requests.exceptions.ReadTimeout,
152             requests.exceptions.TooManyRedirects,
153         ) as e:
154             raise_from(
155                 errors.NetworkLocationResponseTimeout(
156                     "Response timeout: {}".format(url)
157                 ),
158                 e,
159             )
160         except (
161             requests.exceptions.HTTPError,
162             requests.exceptions.ContentDecodingError,
163             requests.exceptions.ChunkedEncodingError,
164             requests.exceptions.RequestException,
165         ) as e:
166             raise_from(
167                 errors.NetworkLocationResponseFailure(
168                     "Response failure: {}".format(url), response=response
169                 ),
170                 e,
171             )
172 
173     def connect(self, raise_if_unavailable=True):  # noqa: C901
174         """
175         Attempts a connection to the instance and caches its device information if successful
176         :param raise_if_unavailable: Raises an error if connection fails and this value is True
177         :return: A boolean determining success, never False if `raise_if_unavailable=True`
178         """
179 
180         from kolibri.core.device.utils import DEVICE_INFO_VERSION
181         from kolibri.core.device.utils import device_info_keys
182 
183         # don't reconnect if client has already done so
184         if self.device_info is not None:
185             return True
186 
187         try:
188             logger.info("Attempting connection to: {}".format(self.base_url))
189             response = self.get(
190                 "api/public/info/",
191                 allow_redirects=True,
192                 params={"v": DEVICE_INFO_VERSION},
193                 timeout=(DEFAULT_CONNECT_TIMEOUT, 5),
194             )
195         except errors.NetworkClientError as e:
196             logger.info(e)
197             if raise_if_unavailable:
198                 raise e
199             return False
200 
201         # check that we successfully connected, and if we were redirected that it's still
202         # the right endpoint
203         parsed_url = urlparse(response.url)
204         if response.status_code != 200:
205             if raise_if_unavailable:
206                 raise errors.NetworkLocationInvalidResponse(
207                     "Response status {}".format(response.status_code)
208                 )
209             return False
210         if not parsed_url.path.rstrip("/").endswith("/api/public/info"):
211             if raise_if_unavailable:
212                 raise errors.NetworkLocationInvalidResponse(
213                     "Request redirected to {}".format(parsed_url.path)
214                 )
215             return False
216 
217         try:
218             info = response.json()
219             self.device_info = {}
220             for key in device_info_keys.get(DEVICE_INFO_VERSION, []):
221                 self.device_info[key] = info.get(key, device_info_defaults.get(key))
222             if self.device_info["application"] not in ["studio", "kolibri"]:
223                 raise errors.NetworkLocationInvalidResponse(
224                     "Server is not running Kolibri or Studio"
225                 )
226             logger.info("Success! We connected to: {}".format(response.url))
227 
228             self.base_url = "{}://{}{}".format(
229                 parsed_url.scheme,
230                 parsed_url.netloc,
231                 parsed_url.path.rstrip("/").replace("api/public/info", ""),
232             )
233         except (requests.exceptions.JSONDecodeError, ValueError) as e:
234             logger.info(
235                 "Invalid JSON returned when attempting to connect to a remote server"
236             )
237             if raise_if_unavailable:
238                 raise_from(
239                     errors.NetworkLocationInvalidResponse("Invalid JSON returned"), e
240                 )
241             return False
242 
243         return True
244
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/kolibri/core/discovery/utils/network/client.py b/kolibri/core/discovery/utils/network/client.py
--- a/kolibri/core/discovery/utils/network/client.py
+++ b/kolibri/core/discovery/utils/network/client.py
@@ -4,6 +4,7 @@
 from six import raise_from
 from six.moves.urllib.parse import urlparse
 
+import kolibri
 from . import errors
 from .urls import get_normalized_url_variations
 from .urls import HTTP_PORTS
@@ -47,6 +48,11 @@
         self.session = None
         self.device_info = None
         self.remote_ip = None
+        self.headers.update(
+            {
+                "User-Agent": get_user_agent(),
+            }
+        )
 
     @classmethod
     def build_for_address(cls, address, timeout=None):
@@ -241,3 +247,9 @@
             return False
 
         return True
+
+
+def get_user_agent():
+    return "Kolibri/{0} python-requests/{1}".format(
+        kolibri.__version__, requests.__version__
+    )

{"golden_diff": "diff --git a/kolibri/core/discovery/utils/network/client.py b/kolibri/core/discovery/utils/network/client.py\n--- a/kolibri/core/discovery/utils/network/client.py\n+++ b/kolibri/core/discovery/utils/network/client.py\n@@ -4,6 +4,7 @@\n from six import raise_from\n from six.moves.urllib.parse import urlparse\n \n+import kolibri\n from . import errors\n from .urls import get_normalized_url_variations\n from .urls import HTTP_PORTS\n@@ -47,6 +48,11 @@\n         self.session = None\n         self.device_info = None\n         self.remote_ip = None\n+        self.headers.update(\n+            {\n+                \"User-Agent\": get_user_agent(),\n+            }\n+        )\n \n     @classmethod\n     def build_for_address(cls, address, timeout=None):\n@@ -241,3 +247,9 @@\n             return False\n \n         return True\n+\n+\n+def get_user_agent():\n+    return \"Kolibri/{0} python-requests/{1}\".format(\n+        kolibri.__version__, requests.__version__\n+    )\n", "issue": "Add custom user-agent to `NetworkClient` and external API calls\n<!--\r\nInstructions:\r\n * Fill out the sections below, replace \u2026's with information about your issue\r\n * Use the 'preview' function above this text box to verify formatting before submitting\r\n-->\r\n\r\n## Observed behavior\r\n<!--\r\nDescription of the behavior that was observed, including screenshots or other references when applicable\r\n-->\r\nThe user agent for API calls made with python `requests` do not clearly identify the device as Kolibri\r\n\r\n## Errors and logs\r\n<!--\r\nRelevant logs from:\r\n * the command line\r\n * ~/.kolibri/logs/kolibri.txt\r\n * the browser console\r\n\r\nPlease wrap errors in triple backticks for clean formatting like this:\r\n```\r\n01:10 info: something happened\r\n01:12 error: something bad happened\r\n```\r\n-->\r\nPerhaps connected with errors like the following although these are not the primary reason to customize the user-agent:\r\n```\r\nhttp.client.RemoteDisconnected: Remote end closed connection without response\r\n```\r\n\r\n## Expected behavior\r\n<!--\r\nDescription of what behavior was expected but did not occur\r\n-->\r\nIt's recommended that we clearly identify the application making API calls, to differentiate it from potential scripting. According to MDN, it should have the format:\r\n```\r\nUser-Agent: <product> / <product-version> <comment>\r\n```\r\nSo for Kolibri 0.16.0:\r\n```\r\nUser-Agent: kolibri/0.16.0 python-requests/2.28.2\r\n```\r\n\r\n## User-facing consequences\r\n<!--\r\nImplications and real-world consequences for learners, coaches, admins, and other users of the application\r\n-->\r\nWeb application firewalls may take more aggressive action against clients making many requests if it appears to be purely something scripted.\r\n\r\n\r\n## Context\r\n<!--\r\nTell us about your environment, including:\r\n * Kolibri version\r\n * Operating system\r\n * Browser\r\n-->\r\nKolibri 0.15+\r\n\n", "before_files": [{"content": "import logging\n\nimport requests\nfrom six import raise_from\nfrom six.moves.urllib.parse import urlparse\n\nfrom . import errors\nfrom .urls import get_normalized_url_variations\nfrom .urls import HTTP_PORTS\nfrom .urls import HTTPS_PORTS\nfrom kolibri.core.discovery.models import ConnectionStatus\nfrom kolibri.core.tasks.utils import get_current_job\nfrom kolibri.core.utils.urls import join_url\nfrom kolibri.utils.server import get_urls\n\nlogger = logging.getLogger(__name__)\n\ndevice_info_defaults = {\n    \"subset_of_users_device\": False,\n}\n\nDEFAULT_CONNECT_TIMEOUT = 5\nDEFAULT_READ_TIMEOUT = 60\n# default read timeout when within a job\nDEFAULT_ASYNC_READ_TIMEOUT = 30\n# when the network client tries variations of a url, that means the overall length of time it takes\n# is multiplied by the number of variations, so for synchronous operations (in a HTTP request) we\n# make the overall timeout ~= the DEFAULT_READ_TIMEOUT\nDEFAULT_SYNC_READ_TIMEOUT = DEFAULT_READ_TIMEOUT / (len(HTTP_PORTS) + len(HTTPS_PORTS))\n\n\nclass NetworkClient(requests.Session):\n    __slots__ = (\"base_url\", \"timeout\", \"session\", \"device_info\", \"remote_ip\")\n\n    def __init__(self, base_url, timeout=None):\n        \"\"\"\n        If an explicit base_url is already known, provide that. If only a vague address is known,\n        `build_from_address` can build a client to determine the actual `base_url`\n        :param base_url: The fully composed URL for a network location, without path\n        :param timeout: A timeout value in seconds or tuple for (connect, read)\n        :type timeout: float|tuple\n        \"\"\"\n        super(NetworkClient, self).__init__()\n\n        self.base_url = base_url\n        self.timeout = timeout or (DEFAULT_CONNECT_TIMEOUT, DEFAULT_READ_TIMEOUT)\n        self.session = None\n        self.device_info = None\n        self.remote_ip = None\n\n    @classmethod\n    def build_for_address(cls, address, timeout=None):\n        \"\"\"\n        Normalizes the address URL and tries a number of variations until we find one\n        that's able to connect\n\n        :param address: The address of which to try variations of\n        :param timeout: A timeout value in seconds or tuple for (connect, read)\n        :return: A NetworkClient with a verified connection\n        :rtype: NetworkClient\n        \"\"\"\n        logger.info(\n            \"Attempting connections to variations of the URL: {}\".format(address)\n        )\n        if timeout is None:\n            if get_current_job() is not None:\n                # when we're within a job, then we can use longer timeouts\n                timeout = (DEFAULT_CONNECT_TIMEOUT, DEFAULT_ASYNC_READ_TIMEOUT)\n            else:\n                # if we're within a request thread, then we limit it for an overall time\n                timeout = (DEFAULT_CONNECT_TIMEOUT, DEFAULT_SYNC_READ_TIMEOUT)\n        _, self_urls = get_urls()\n        for url in get_normalized_url_variations(address):\n            if url in self_urls:\n                continue  # exclude our own URLs\n            with NetworkClient(url, timeout=timeout) as client:\n                if client.connect(raise_if_unavailable=False):\n                    return client\n        # we weren't able to connect to any of the URL variations, so all we can do is throw\n        raise errors.NetworkLocationNotFound()\n\n    @classmethod\n    def build_from_network_location(cls, network_location, timeout=None):\n        \"\"\"\n        Creates a NetworkClient for a NetworkLocation, and validates the connection if the status\n        isn't already 'Okay'\n        :param network_location: The network location model\n        :type network_location: kolibri.core.discovery.models.NetworkLocation\n        :param timeout: A timeout value in seconds or tuple for (connect, read)\n        :return: A NetworkClient with a verified connection\n        :rtype: NetworkClient\n        \"\"\"\n        # expect that static network locations have an exact base_url, and only try different\n        # variations if we haven't already\n        if (\n            network_location.dynamic\n            and network_location.connection_status == ConnectionStatus.Unknown\n        ):\n            return NetworkClient.build_for_address(\n                network_location.base_url, timeout=timeout\n            )\n        return NetworkClient(network_location.base_url, timeout=timeout)\n\n    def head(self, path, **kwargs):\n        return self.request(\"HEAD\", path, **kwargs)\n\n    def get(self, path, **kwargs):\n        return self.request(\"GET\", path, **kwargs)\n\n    def post(self, path, **kwargs):\n        return self.request(\"POST\", path, **kwargs)\n\n    def request(self, method, path, **kwargs):\n        response = None\n        if \"timeout\" not in kwargs:\n            kwargs.update(timeout=self.timeout)\n\n        url = join_url(self.base_url, path)\n        try:\n            with super(NetworkClient, self).request(\n                method, url, stream=True, **kwargs\n            ) as response:\n                if response.raw._connection.sock is None:\n                    raise requests.exceptions.ConnectionError(\"No socket available\")\n\n                # capture the remote IP address, which requires `stream=True` and before consumed\n                self.remote_ip = response.raw._connection.sock.getpeername()[0]\n                # now consume content, see how `Session.send` does this when `stream=False`\n                response.content\n\n            response.raise_for_status()\n            return response\n        except (\n            requests.exceptions.ConnectionError,\n            requests.exceptions.SSLError,\n            requests.exceptions.ConnectTimeout,\n            requests.exceptions.URLRequired,\n            requests.exceptions.MissingSchema,\n            requests.exceptions.InvalidSchema,\n            requests.exceptions.InvalidURL,\n            requests.exceptions.InvalidHeader,\n            requests.exceptions.InvalidJSONError,\n        ) as e:\n            raise_from(\n                errors.NetworkLocationConnectionFailure(\n                    \"Unable to connect: {}\".format(url)\n                ),\n                e,\n            )\n        except (\n            requests.exceptions.ReadTimeout,\n            requests.exceptions.TooManyRedirects,\n        ) as e:\n            raise_from(\n                errors.NetworkLocationResponseTimeout(\n                    \"Response timeout: {}\".format(url)\n                ),\n                e,\n            )\n        except (\n            requests.exceptions.HTTPError,\n            requests.exceptions.ContentDecodingError,\n            requests.exceptions.ChunkedEncodingError,\n            requests.exceptions.RequestException,\n        ) as e:\n            raise_from(\n                errors.NetworkLocationResponseFailure(\n                    \"Response failure: {}\".format(url), response=response\n                ),\n                e,\n            )\n\n    def connect(self, raise_if_unavailable=True):  # noqa: C901\n        \"\"\"\n        Attempts a connection to the instance and caches its device information if successful\n        :param raise_if_unavailable: Raises an error if connection fails and this value is True\n        :return: A boolean determining success, never False if `raise_if_unavailable=True`\n        \"\"\"\n\n        from kolibri.core.device.utils import DEVICE_INFO_VERSION\n        from kolibri.core.device.utils import device_info_keys\n\n        # don't reconnect if client has already done so\n        if self.device_info is not None:\n            return True\n\n        try:\n            logger.info(\"Attempting connection to: {}\".format(self.base_url))\n            response = self.get(\n                \"api/public/info/\",\n                allow_redirects=True,\n                params={\"v\": DEVICE_INFO_VERSION},\n                timeout=(DEFAULT_CONNECT_TIMEOUT, 5),\n            )\n        except errors.NetworkClientError as e:\n            logger.info(e)\n            if raise_if_unavailable:\n                raise e\n            return False\n\n        # check that we successfully connected, and if we were redirected that it's still\n        # the right endpoint\n        parsed_url = urlparse(response.url)\n        if response.status_code != 200:\n            if raise_if_unavailable:\n                raise errors.NetworkLocationInvalidResponse(\n                    \"Response status {}\".format(response.status_code)\n                )\n            return False\n        if not parsed_url.path.rstrip(\"/\").endswith(\"/api/public/info\"):\n            if raise_if_unavailable:\n                raise errors.NetworkLocationInvalidResponse(\n                    \"Request redirected to {}\".format(parsed_url.path)\n                )\n            return False\n\n        try:\n            info = response.json()\n            self.device_info = {}\n            for key in device_info_keys.get(DEVICE_INFO_VERSION, []):\n                self.device_info[key] = info.get(key, device_info_defaults.get(key))\n            if self.device_info[\"application\"] not in [\"studio\", \"kolibri\"]:\n                raise errors.NetworkLocationInvalidResponse(\n                    \"Server is not running Kolibri or Studio\"\n                )\n            logger.info(\"Success! We connected to: {}\".format(response.url))\n\n            self.base_url = \"{}://{}{}\".format(\n                parsed_url.scheme,\n                parsed_url.netloc,\n                parsed_url.path.rstrip(\"/\").replace(\"api/public/info\", \"\"),\n            )\n        except (requests.exceptions.JSONDecodeError, ValueError) as e:\n            logger.info(\n                \"Invalid JSON returned when attempting to connect to a remote server\"\n            )\n            if raise_if_unavailable:\n                raise_from(\n                    errors.NetworkLocationInvalidResponse(\"Invalid JSON returned\"), e\n                )\n            return False\n\n        return True\n", "path": "kolibri/core/discovery/utils/network/client.py"}], "after_files": [{"content": "import logging\n\nimport requests\nfrom six import raise_from\nfrom six.moves.urllib.parse import urlparse\n\nimport kolibri\nfrom . import errors\nfrom .urls import get_normalized_url_variations\nfrom .urls import HTTP_PORTS\nfrom .urls import HTTPS_PORTS\nfrom kolibri.core.discovery.models import ConnectionStatus\nfrom kolibri.core.tasks.utils import get_current_job\nfrom kolibri.core.utils.urls import join_url\nfrom kolibri.utils.server import get_urls\n\nlogger = logging.getLogger(__name__)\n\ndevice_info_defaults = {\n    \"subset_of_users_device\": False,\n}\n\nDEFAULT_CONNECT_TIMEOUT = 5\nDEFAULT_READ_TIMEOUT = 60\n# default read timeout when within a job\nDEFAULT_ASYNC_READ_TIMEOUT = 30\n# when the network client tries variations of a url, that means the overall length of time it takes\n# is multiplied by the number of variations, so for synchronous operations (in a HTTP request) we\n# make the overall timeout ~= the DEFAULT_READ_TIMEOUT\nDEFAULT_SYNC_READ_TIMEOUT = DEFAULT_READ_TIMEOUT / (len(HTTP_PORTS) + len(HTTPS_PORTS))\n\n\nclass NetworkClient(requests.Session):\n    __slots__ = (\"base_url\", \"timeout\", \"session\", \"device_info\", \"remote_ip\")\n\n    def __init__(self, base_url, timeout=None):\n        \"\"\"\n        If an explicit base_url is already known, provide that. If only a vague address is known,\n        `build_from_address` can build a client to determine the actual `base_url`\n        :param base_url: The fully composed URL for a network location, without path\n        :param timeout: A timeout value in seconds or tuple for (connect, read)\n        :type timeout: float|tuple\n        \"\"\"\n        super(NetworkClient, self).__init__()\n\n        self.base_url = base_url\n        self.timeout = timeout or (DEFAULT_CONNECT_TIMEOUT, DEFAULT_READ_TIMEOUT)\n        self.session = None\n        self.device_info = None\n        self.remote_ip = None\n        self.headers.update(\n            {\n                \"User-Agent\": get_user_agent(),\n            }\n        )\n\n    @classmethod\n    def build_for_address(cls, address, timeout=None):\n        \"\"\"\n        Normalizes the address URL and tries a number of variations until we find one\n        that's able to connect\n\n        :param address: The address of which to try variations of\n        :param timeout: A timeout value in seconds or tuple for (connect, read)\n        :return: A NetworkClient with a verified connection\n        :rtype: NetworkClient\n        \"\"\"\n        logger.info(\n            \"Attempting connections to variations of the URL: {}\".format(address)\n        )\n        if timeout is None:\n            if get_current_job() is not None:\n                # when we're within a job, then we can use longer timeouts\n                timeout = (DEFAULT_CONNECT_TIMEOUT, DEFAULT_ASYNC_READ_TIMEOUT)\n            else:\n                # if we're within a request thread, then we limit it for an overall time\n                timeout = (DEFAULT_CONNECT_TIMEOUT, DEFAULT_SYNC_READ_TIMEOUT)\n        _, self_urls = get_urls()\n        for url in get_normalized_url_variations(address):\n            if url in self_urls:\n                continue  # exclude our own URLs\n            with NetworkClient(url, timeout=timeout) as client:\n                if client.connect(raise_if_unavailable=False):\n                    return client\n        # we weren't able to connect to any of the URL variations, so all we can do is throw\n        raise errors.NetworkLocationNotFound()\n\n    @classmethod\n    def build_from_network_location(cls, network_location, timeout=None):\n        \"\"\"\n        Creates a NetworkClient for a NetworkLocation, and validates the connection if the status\n        isn't already 'Okay'\n        :param network_location: The network location model\n        :type network_location: kolibri.core.discovery.models.NetworkLocation\n        :param timeout: A timeout value in seconds or tuple for (connect, read)\n        :return: A NetworkClient with a verified connection\n        :rtype: NetworkClient\n        \"\"\"\n        # expect that static network locations have an exact base_url, and only try different\n        # variations if we haven't already\n        if (\n            network_location.dynamic\n            and network_location.connection_status == ConnectionStatus.Unknown\n        ):\n            return NetworkClient.build_for_address(\n                network_location.base_url, timeout=timeout\n            )\n        return NetworkClient(network_location.base_url, timeout=timeout)\n\n    def head(self, path, **kwargs):\n        return self.request(\"HEAD\", path, **kwargs)\n\n    def get(self, path, **kwargs):\n        return self.request(\"GET\", path, **kwargs)\n\n    def post(self, path, **kwargs):\n        return self.request(\"POST\", path, **kwargs)\n\n    def request(self, method, path, **kwargs):\n        response = None\n        if \"timeout\" not in kwargs:\n            kwargs.update(timeout=self.timeout)\n\n        url = join_url(self.base_url, path)\n        try:\n            with super(NetworkClient, self).request(\n                method, url, stream=True, **kwargs\n            ) as response:\n                if response.raw._connection.sock is None:\n                    raise requests.exceptions.ConnectionError(\"No socket available\")\n\n                # capture the remote IP address, which requires `stream=True` and before consumed\n                self.remote_ip = response.raw._connection.sock.getpeername()[0]\n                # now consume content, see how `Session.send` does this when `stream=False`\n                response.content\n\n            response.raise_for_status()\n            return response\n        except (\n            requests.exceptions.ConnectionError,\n            requests.exceptions.SSLError,\n            requests.exceptions.ConnectTimeout,\n            requests.exceptions.URLRequired,\n            requests.exceptions.MissingSchema,\n            requests.exceptions.InvalidSchema,\n            requests.exceptions.InvalidURL,\n            requests.exceptions.InvalidHeader,\n            requests.exceptions.InvalidJSONError,\n        ) as e:\n            raise_from(\n                errors.NetworkLocationConnectionFailure(\n                    \"Unable to connect: {}\".format(url)\n                ),\n                e,\n            )\n        except (\n            requests.exceptions.ReadTimeout,\n            requests.exceptions.TooManyRedirects,\n        ) as e:\n            raise_from(\n                errors.NetworkLocationResponseTimeout(\n                    \"Response timeout: {}\".format(url)\n                ),\n                e,\n            )\n        except (\n            requests.exceptions.HTTPError,\n            requests.exceptions.ContentDecodingError,\n            requests.exceptions.ChunkedEncodingError,\n            requests.exceptions.RequestException,\n        ) as e:\n            raise_from(\n                errors.NetworkLocationResponseFailure(\n                    \"Response failure: {}\".format(url), response=response\n                ),\n                e,\n            )\n\n    def connect(self, raise_if_unavailable=True):  # noqa: C901\n        \"\"\"\n        Attempts a connection to the instance and caches its device information if successful\n        :param raise_if_unavailable: Raises an error if connection fails and this value is True\n        :return: A boolean determining success, never False if `raise_if_unavailable=True`\n        \"\"\"\n\n        from kolibri.core.device.utils import DEVICE_INFO_VERSION\n        from kolibri.core.device.utils import device_info_keys\n\n        # don't reconnect if client has already done so\n        if self.device_info is not None:\n            return True\n\n        try:\n            logger.info(\"Attempting connection to: {}\".format(self.base_url))\n            response = self.get(\n                \"api/public/info/\",\n                allow_redirects=True,\n                params={\"v\": DEVICE_INFO_VERSION},\n                timeout=(DEFAULT_CONNECT_TIMEOUT, 5),\n            )\n        except errors.NetworkClientError as e:\n            logger.info(e)\n            if raise_if_unavailable:\n                raise e\n            return False\n\n        # check that we successfully connected, and if we were redirected that it's still\n        # the right endpoint\n        parsed_url = urlparse(response.url)\n        if response.status_code != 200:\n            if raise_if_unavailable:\n                raise errors.NetworkLocationInvalidResponse(\n                    \"Response status {}\".format(response.status_code)\n                )\n            return False\n        if not parsed_url.path.rstrip(\"/\").endswith(\"/api/public/info\"):\n            if raise_if_unavailable:\n                raise errors.NetworkLocationInvalidResponse(\n                    \"Request redirected to {}\".format(parsed_url.path)\n                )\n            return False\n\n        try:\n            info = response.json()\n            self.device_info = {}\n            for key in device_info_keys.get(DEVICE_INFO_VERSION, []):\n                self.device_info[key] = info.get(key, device_info_defaults.get(key))\n            if self.device_info[\"application\"] not in [\"studio\", \"kolibri\"]:\n                raise errors.NetworkLocationInvalidResponse(\n                    \"Server is not running Kolibri or Studio\"\n                )\n            logger.info(\"Success! We connected to: {}\".format(response.url))\n\n            self.base_url = \"{}://{}{}\".format(\n                parsed_url.scheme,\n                parsed_url.netloc,\n                parsed_url.path.rstrip(\"/\").replace(\"api/public/info\", \"\"),\n            )\n        except (requests.exceptions.JSONDecodeError, ValueError) as e:\n            logger.info(\n                \"Invalid JSON returned when attempting to connect to a remote server\"\n            )\n            if raise_if_unavailable:\n                raise_from(\n                    errors.NetworkLocationInvalidResponse(\"Invalid JSON returned\"), e\n                )\n            return False\n\n        return True\n\n\ndef get_user_agent():\n    return \"Kolibri/{0} python-requests/{1}\".format(\n        kolibri.__version__, requests.__version__\n    )\n", "path": "kolibri/core/discovery/utils/network/client.py"}]}