We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[BUG] Service metrics endpoint excludes many important routes
### Willingness to contribute
The MLflow Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the MLflow code base?

- [x] Yes. I can contribute a fix for this bug independently.
- [ ] Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.
- [ ] No. I cannot contribute a bug fix at this time.

### System information
- **Have I written custom code (as opposed to using a stock example script provided in MLflow)**: no
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Debian GNU/Linux 11
- **MLflow installed from (source or binary)**: binary
- **MLflow version (run ``mlflow --version``)**: 1.21.0
- **Python version**: 3.8
- **npm version, if running the dev UI**:
- **Exact command to reproduce**: see section below

### Describe the problem

The mlflow server option to expose a Prometheus metrics endpoint is a great observability feature for MLflow. Unfortunately, the current implementation leaves an incomplete view of the server health/performance. Currently, mlflow only logs metrics to a [subset of endpoints](https://github.com/mlflow/mlflow/blob/master/mlflow/server/prometheus_exporter.py#L18).

As of MLflow version 1.21, the following routes are not being included in the service metrics:

```
['static', '_get_experiment_by_name', '_create_experiment', '_list_experiments', '_get_experiment',
'_delete_experiment', '_restore_experiment', '_update_experiment',  '_update_run', '_delete_run', 
'_restore_run', '_set_experiment_tag',  '_delete_tag', '_get_run',  '_list_artifacts', '_get_metric_history',
'_log_batch', '_log_model', '_create_registered_model', '_rename_registered_model', 
'_update_registered_model',  '_delete_registered_model', '_get_registered_model', '_search_registered_models', 
'_list_registered_models',  '_get_latest_versions', '_create_model_version', '_update_model_version', 
'_transition_stage', '_delete_model_version',  '_get_model_version', '_search_model_versions', 
'_get_model_version_download_uri', '_set_registered_model_tag', '_set_model_version_tag', 
'_delete_registered_model_tag', '_delete_model_version_tag', 'health', 'serve_artifacts', 
'serve_model_version_artifact', 'serve_static_file', 'serve']
```

(see full list of endpoints)
```
from mlflow.server import app

app.view_functions.keys()
```

Filtering the set of routes to be included in the metrics endpoint seems like a potentially fragile approach as new routes are added in later versions of mlflow. It's especially problematic that the list of filtered routes cannot be configured. We currently have no way to monitor the health of the overall service given that many key routes (e.g. `log_batch`) are not included in the service metrics.

### Code to reproduce issue

Dockerfile for mlflow server
```
FROM python:3.8
RUN pip install mlflow==1.21.0

ENTRYPOINT mlflow server \
    --backend-store-uri sqlite:///mlflow.sqlite \
    --default-artifact-root file:///artifacts \
    --host 0.0.0.0 \
    --port 5000 \
    --expose-prometheus /prometheus
```

Build and run the Docker container
```
docker build -t mlflow_example -f Dockerfile .
docker run -p 5000:5000 mlflow_example
```


Script with incomplete representation in metrics endpoint
```
import mlflow
import random

mlflow.set_tracking_uri("http://127.0.0.1:5000")
mlflow.set_experiment("service_metrics")

with mlflow.start_run(run_name="test"):

    for _ in range(100):
        mlflow.log_metrics({
            'loss_a': random.random(),
            'loss_b': random.random(),
            'loss_c': random.random(),
        })

    mlflow.log_params({'a': 1, 'b': 2, 'c': 3})
```
See how metrics for these endpoints **_do not_** appear at http://127.0.0.1:5000/metrics

---

Script with expected representation in metrics endpoint
```
import mlflow
import random

mlflow.set_tracking_uri("http://127.0.0.1:5000")
mlflow.set_experiment("service_metrics")

with mlflow.start_run(run_name="test"):
    for _ in range(100):
        mlflow.log_metric('loss', random.random())

    mlflow.log_param('param', 'test')
```
See how metrics for these endpoints appear at http://127.0.0.1:5000/metrics

### Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.


### What component(s), interfaces, languages, and integrations does this bug affect?
Components 
- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [x] `area/server-infra`: MLflow Tracking server backend
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface 
- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Language 
- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations
- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `mlflow/server/prometheus_exporter.py`
Content:
```
1 from prometheus_flask_exporter.multiprocess import GunicornInternalPrometheusMetrics
2 from flask import request
3 
4 
5 def activate_prometheus_exporter(app):
6     metrics = GunicornInternalPrometheusMetrics(app, export_defaults=False)
7 
8     endpoint = app.view_functions
9     histogram = metrics.histogram(
10         "mlflow_requests_by_status_and_path",
11         "Request latencies and count by status and path",
12         labels={
13             "status": lambda r: r.status_code,
14             "path": lambda: change_path_for_metric(request.path),
15         },
16     )
17     for func_name, func in endpoint.items():
18         if func_name in ["_search_runs", "_log_metric", "_log_param", "_set_tag", "_create_run"]:
19             app.view_functions[func_name] = histogram(func)
20 
21     return app
22 
23 
24 def change_path_for_metric(path):
25     """
26     Replace the '/' in the metric path by '_' so grafana can correctly use it.
27     :param path: path of the metric (example: runs/search)
28     :return: path with '_' instead of '/'
29     """
30     if "mlflow/" in path:
31         path = path.split("mlflow/")[-1]
32     return path.replace("/", "_")
33
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/mlflow/server/prometheus_exporter.py b/mlflow/server/prometheus_exporter.py
--- a/mlflow/server/prometheus_exporter.py
+++ b/mlflow/server/prometheus_exporter.py
@@ -1,32 +1,19 @@
 from prometheus_flask_exporter.multiprocess import GunicornInternalPrometheusMetrics
 from flask import request
 
+from mlflow.version import VERSION
+
 
 def activate_prometheus_exporter(app):
-    metrics = GunicornInternalPrometheusMetrics(app, export_defaults=False)
+    def mlflow_version(_: request):
+        return VERSION
 
-    endpoint = app.view_functions
-    histogram = metrics.histogram(
-        "mlflow_requests_by_status_and_path",
-        "Request latencies and count by status and path",
-        labels={
-            "status": lambda r: r.status_code,
-            "path": lambda: change_path_for_metric(request.path),
-        },
+    metrics = GunicornInternalPrometheusMetrics(
+        app,
+        export_defaults=True,
+        defaults_prefix="mlflow",
+        excluded_paths=["/health"],
+        group_by=mlflow_version,
     )
-    for func_name, func in endpoint.items():
-        if func_name in ["_search_runs", "_log_metric", "_log_param", "_set_tag", "_create_run"]:
-            app.view_functions[func_name] = histogram(func)
-
-    return app
-
 
-def change_path_for_metric(path):
-    """
-    Replace the '/' in the metric path by '_' so grafana can correctly use it.
-    :param path: path of the metric (example: runs/search)
-    :return: path with '_' instead of '/'
-    """
-    if "mlflow/" in path:
-        path = path.split("mlflow/")[-1]
-    return path.replace("/", "_")
+    return metrics

{"golden_diff": "diff --git a/mlflow/server/prometheus_exporter.py b/mlflow/server/prometheus_exporter.py\n--- a/mlflow/server/prometheus_exporter.py\n+++ b/mlflow/server/prometheus_exporter.py\n@@ -1,32 +1,19 @@\n from prometheus_flask_exporter.multiprocess import GunicornInternalPrometheusMetrics\r\n from flask import request\r\n \r\n+from mlflow.version import VERSION\r\n+\r\n \r\n def activate_prometheus_exporter(app):\r\n-    metrics = GunicornInternalPrometheusMetrics(app, export_defaults=False)\r\n+    def mlflow_version(_: request):\r\n+        return VERSION\r\n \r\n-    endpoint = app.view_functions\r\n-    histogram = metrics.histogram(\r\n-        \"mlflow_requests_by_status_and_path\",\r\n-        \"Request latencies and count by status and path\",\r\n-        labels={\r\n-            \"status\": lambda r: r.status_code,\r\n-            \"path\": lambda: change_path_for_metric(request.path),\r\n-        },\r\n+    metrics = GunicornInternalPrometheusMetrics(\r\n+        app,\r\n+        export_defaults=True,\r\n+        defaults_prefix=\"mlflow\",\r\n+        excluded_paths=[\"/health\"],\r\n+        group_by=mlflow_version,\r\n     )\r\n-    for func_name, func in endpoint.items():\r\n-        if func_name in [\"_search_runs\", \"_log_metric\", \"_log_param\", \"_set_tag\", \"_create_run\"]:\r\n-            app.view_functions[func_name] = histogram(func)\r\n-\r\n-    return app\r\n-\r\n \r\n-def change_path_for_metric(path):\r\n-    \"\"\"\r\n-    Replace the '/' in the metric path by '_' so grafana can correctly use it.\r\n-    :param path: path of the metric (example: runs/search)\r\n-    :return: path with '_' instead of '/'\r\n-    \"\"\"\r\n-    if \"mlflow/\" in path:\r\n-        path = path.split(\"mlflow/\")[-1]\r\n-    return path.replace(\"/\", \"_\")\r\n+    return metrics\n", "issue": "[BUG] Service metrics endpoint excludes many important routes\n### Willingness to contribute\r\nThe MLflow Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the MLflow code base?\r\n\r\n- [x] Yes. I can contribute a fix for this bug independently.\r\n- [ ] Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.\r\n- [ ] No. I cannot contribute a bug fix at this time.\r\n\r\n### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in MLflow)**: no\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Debian GNU/Linux 11\r\n- **MLflow installed from (source or binary)**: binary\r\n- **MLflow version (run ``mlflow --version``)**: 1.21.0\r\n- **Python version**: 3.8\r\n- **npm version, if running the dev UI**:\r\n- **Exact command to reproduce**: see section below\r\n\r\n### Describe the problem\r\n\r\nThe mlflow server option to expose a Prometheus metrics endpoint is a great observability feature for MLflow. Unfortunately, the current implementation leaves an incomplete view of the server health/performance. Currently, mlflow only logs metrics to a [subset of endpoints](https://github.com/mlflow/mlflow/blob/master/mlflow/server/prometheus_exporter.py#L18).\r\n\r\nAs of MLflow version 1.21, the following routes are not being included in the service metrics:\r\n\r\n```\r\n['static', '_get_experiment_by_name', '_create_experiment', '_list_experiments', '_get_experiment',\r\n'_delete_experiment', '_restore_experiment', '_update_experiment',  '_update_run', '_delete_run', \r\n'_restore_run', '_set_experiment_tag',  '_delete_tag', '_get_run',  '_list_artifacts', '_get_metric_history',\r\n'_log_batch', '_log_model', '_create_registered_model', '_rename_registered_model', \r\n'_update_registered_model',  '_delete_registered_model', '_get_registered_model', '_search_registered_models', \r\n'_list_registered_models',  '_get_latest_versions', '_create_model_version', '_update_model_version', \r\n'_transition_stage', '_delete_model_version',  '_get_model_version', '_search_model_versions', \r\n'_get_model_version_download_uri', '_set_registered_model_tag', '_set_model_version_tag', \r\n'_delete_registered_model_tag', '_delete_model_version_tag', 'health', 'serve_artifacts', \r\n'serve_model_version_artifact', 'serve_static_file', 'serve']\r\n```\r\n\r\n(see full list of endpoints)\r\n```\r\nfrom mlflow.server import app\r\n\r\napp.view_functions.keys()\r\n```\r\n\r\nFiltering the set of routes to be included in the metrics endpoint seems like a potentially fragile approach as new routes are added in later versions of mlflow. It's especially problematic that the list of filtered routes cannot be configured. We currently have no way to monitor the health of the overall service given that many key routes (e.g. `log_batch`) are not included in the service metrics.\r\n\r\n### Code to reproduce issue\r\n\r\nDockerfile for mlflow server\r\n```\r\nFROM python:3.8\r\nRUN pip install mlflow==1.21.0\r\n\r\nENTRYPOINT mlflow server \\\r\n    --backend-store-uri sqlite:///mlflow.sqlite \\\r\n    --default-artifact-root file:///artifacts \\\r\n    --host 0.0.0.0 \\\r\n    --port 5000 \\\r\n    --expose-prometheus /prometheus\r\n```\r\n\r\nBuild and run the Docker container\r\n```\r\ndocker build -t mlflow_example -f Dockerfile .\r\ndocker run -p 5000:5000 mlflow_example\r\n```\r\n\r\n\r\nScript with incomplete representation in metrics endpoint\r\n```\r\nimport mlflow\r\nimport random\r\n\r\nmlflow.set_tracking_uri(\"http://127.0.0.1:5000\")\r\nmlflow.set_experiment(\"service_metrics\")\r\n\r\nwith mlflow.start_run(run_name=\"test\"):\r\n\r\n    for _ in range(100):\r\n        mlflow.log_metrics({\r\n            'loss_a': random.random(),\r\n            'loss_b': random.random(),\r\n            'loss_c': random.random(),\r\n        })\r\n\r\n    mlflow.log_params({'a': 1, 'b': 2, 'c': 3})\r\n```\r\nSee how metrics for these endpoints **_do not_** appear at http://127.0.0.1:5000/metrics\r\n\r\n---\r\n\r\nScript with expected representation in metrics endpoint\r\n```\r\nimport mlflow\r\nimport random\r\n\r\nmlflow.set_tracking_uri(\"http://127.0.0.1:5000\")\r\nmlflow.set_experiment(\"service_metrics\")\r\n\r\nwith mlflow.start_run(run_name=\"test\"):\r\n    for _ in range(100):\r\n        mlflow.log_metric('loss', random.random())\r\n\r\n    mlflow.log_param('param', 'test')\r\n```\r\nSee how metrics for these endpoints appear at http://127.0.0.1:5000/metrics\r\n\r\n### Other info / logs\r\nInclude any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.\r\n\r\n\r\n### What component(s), interfaces, languages, and integrations does this bug affect?\r\nComponents \r\n- [ ] `area/artifacts`: Artifact stores and artifact logging\r\n- [ ] `area/build`: Build and test infrastructure for MLflow\r\n- [ ] `area/docs`: MLflow documentation pages\r\n- [ ] `area/examples`: Example code\r\n- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry\r\n- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors\r\n- [ ] `area/projects`: MLproject format, project running backends\r\n- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs\r\n- [x] `area/server-infra`: MLflow Tracking server backend\r\n- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging\r\n\r\nInterface \r\n- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server\r\n- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models\r\n- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry\r\n- [ ] `area/windows`: Windows support\r\n\r\nLanguage \r\n- [ ] `language/r`: R APIs and clients\r\n- [ ] `language/java`: Java APIs and clients\r\n- [ ] `language/new`: Proposals for new client languages\r\n\r\nIntegrations\r\n- [ ] `integrations/azure`: Azure and Azure ML integrations\r\n- [ ] `integrations/sagemaker`: SageMaker integrations\r\n- [ ] `integrations/databricks`: Databricks integrations\r\n\n", "before_files": [{"content": "from prometheus_flask_exporter.multiprocess import GunicornInternalPrometheusMetrics\r\nfrom flask import request\r\n\r\n\r\ndef activate_prometheus_exporter(app):\r\n    metrics = GunicornInternalPrometheusMetrics(app, export_defaults=False)\r\n\r\n    endpoint = app.view_functions\r\n    histogram = metrics.histogram(\r\n        \"mlflow_requests_by_status_and_path\",\r\n        \"Request latencies and count by status and path\",\r\n        labels={\r\n            \"status\": lambda r: r.status_code,\r\n            \"path\": lambda: change_path_for_metric(request.path),\r\n        },\r\n    )\r\n    for func_name, func in endpoint.items():\r\n        if func_name in [\"_search_runs\", \"_log_metric\", \"_log_param\", \"_set_tag\", \"_create_run\"]:\r\n            app.view_functions[func_name] = histogram(func)\r\n\r\n    return app\r\n\r\n\r\ndef change_path_for_metric(path):\r\n    \"\"\"\r\n    Replace the '/' in the metric path by '_' so grafana can correctly use it.\r\n    :param path: path of the metric (example: runs/search)\r\n    :return: path with '_' instead of '/'\r\n    \"\"\"\r\n    if \"mlflow/\" in path:\r\n        path = path.split(\"mlflow/\")[-1]\r\n    return path.replace(\"/\", \"_\")\r\n", "path": "mlflow/server/prometheus_exporter.py"}], "after_files": [{"content": "from prometheus_flask_exporter.multiprocess import GunicornInternalPrometheusMetrics\r\nfrom flask import request\r\n\r\nfrom mlflow.version import VERSION\r\n\r\n\r\ndef activate_prometheus_exporter(app):\r\n    def mlflow_version(_: request):\r\n        return VERSION\r\n\r\n    metrics = GunicornInternalPrometheusMetrics(\r\n        app,\r\n        export_defaults=True,\r\n        defaults_prefix=\"mlflow\",\r\n        excluded_paths=[\"/health\"],\r\n        group_by=mlflow_version,\r\n    )\r\n\r\n    return metrics\r\n", "path": "mlflow/server/prometheus_exporter.py"}]}