We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Clean up previously generated CSV files
<!--
Instructions:
 * Fill out the sections below, replace …'s with information about your issue
 * Use the 'preview' function above this text box to verify formatting before submitting
-->

## Observed behavior
When a new CSV file is generated for session and summary logs, the previously generated files that are stored in the user's database are no longer accessible for download in the UI. A record of the new CSV file request is saved to the logger model `GenerateCSVLogRequest` which contains the facility id, log type (session or summary), the selected start date, the selected end date, and the date that the request was made. There should only ever be one record for each facility and log type combination. The information in this modal is used to scan the database folder `log_exports` and retrieve the previously generated file for download.

Note: For session and summary log files, the file naming convention has been updated to include a date-range selection, with the dates formatted as `YYYY-MM-DD`.
Previous naming convention:
- `{facility_name}_{last 4 digits of facility ID}_{log export type}.csv`

Current naming convention:
- `{facility_name}_{last 4 digits of facility ID}_{log export type}_from_{start date}_to_{end date}.csv`

## Expected behavior
We should remove any files that do not have a record in the database. One way to achieve this is by scanning the folder `log_export`, which contains all exported log files, using the predictable file naming convention, and cross-referencing them with the file records stored in the database. Any session or summary files that do not have a corresponding record in the database should be deleted.

With a 1:1 correlation with the `GenerateCSVLogRequest` record, the database should only contain one exported file for each session and summary log type. A new async task should be created for performing this cleanup, and can be queued whenever a new log export is generated. If the task has a predictable id, it can be ensured that there is only one cleanup task going at once.

More details on the task system can be found [here](https://kolibri-dev.readthedocs.io/en/develop/backend_architecture/tasks.html).

## User-facing consequences
Ensures the database is free from redundant files and optimizes the user's database storage.

## Acceptance Criteria

- [ ] All previously generated session and summary log file exports that lack a corresponding record in the database have been deleted.
- [ ] The previously generated CSV file should be automatically removed from the database upon generating a new one.
- [ ] New python code should be covered by unit tests.


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `kolibri/core/logger/tasks.py`
Content:
```
1 import os
2 
3 from django.core.management import call_command
4 from rest_framework import serializers
5 
6 from kolibri.core.auth.models import Facility
7 from kolibri.core.logger.csv_export import CSV_EXPORT_FILENAMES
8 from kolibri.core.tasks.decorators import register_task
9 from kolibri.core.tasks.permissions import IsAdminForJob
10 from kolibri.core.tasks.validation import JobValidator
11 from kolibri.utils import conf
12 
13 
14 def get_filepath(log_type, facility_id, start_date, end_date):
15     facility = Facility.objects.get(id=facility_id)
16     logs_dir = os.path.join(conf.KOLIBRI_HOME, "log_export")
17     if not os.path.isdir(logs_dir):
18         os.mkdir(logs_dir)
19     filepath = os.path.join(
20         logs_dir,
21         CSV_EXPORT_FILENAMES[log_type].format(
22             facility.name, facility.id[:4], start_date[:10], end_date[:10]
23         ),
24     )
25     return filepath
26 
27 
28 class ExportLogCSVValidator(JobValidator):
29     facility = serializers.PrimaryKeyRelatedField(
30         queryset=Facility.objects.all(), required=False
31     )
32     start_date = serializers.CharField()
33     end_date = serializers.CharField()
34     locale = serializers.CharField(required=False)
35 
36     def validate(self, data):
37         facility = data.get("facility", None)
38         start_date = data.get("start_date", None)
39         end_date = data.get("end_date", None)
40         locale = data.get("locale", None)
41 
42         if facility is None and "user" in self.context:
43             facility = self.context["user"].facility
44         elif facility is None:
45             raise serializers.ValidationError(
46                 "Facility must be specified when no user is available."
47             )
48         if not start_date or not end_date:
49             raise serializers.ValidationError(
50                 "Start {} and End {} date values are required.".format(
51                     start_date, end_date
52                 )
53             )
54         kwargs = {
55             "facility": facility.id,
56             "start_date": start_date,
57             "end_date": end_date,
58             "locale": locale,
59         }
60         return {
61             "facility_id": facility.id,
62             "kwargs": kwargs,
63             "args": [facility.id],
64         }
65 
66 
67 def _exportlogcsv(log_type, facility_id, start_date, end_date, locale):
68     filepath = get_filepath(log_type, facility_id, start_date, end_date)
69     call_command(
70         "exportlogs",
71         log_type=log_type,
72         output_file=filepath,
73         facility=facility_id,
74         overwrite=True,
75         start_date=start_date,
76         end_date=end_date,
77         locale=locale,
78     )
79 
80 
81 @register_task(
82     validator=ExportLogCSVValidator,
83     track_progress=True,
84     permission_classes=[IsAdminForJob],
85 )
86 def exportsessionlogcsv(facility_id, **kwargs):
87     """
88     Dumps in csv format the content session logs.
89 
90     :param: facility.
91     """
92     _exportlogcsv(
93         "session",
94         facility_id,
95         kwargs.get("start_date"),
96         kwargs.get("end_date"),
97         kwargs.get("locale"),
98     )
99 
100 
101 @register_task(
102     validator=ExportLogCSVValidator,
103     track_progress=True,
104     permission_classes=[IsAdminForJob],
105 )
106 def exportsummarylogcsv(facility_id, **kwargs):
107     """
108     Dumps in csv format the content summary logs.
109 
110     :param: facility.
111     """
112     _exportlogcsv(
113         "summary",
114         facility_id,
115         kwargs.get("start_date"),
116         kwargs.get("end_date"),
117         kwargs.get("locale"),
118     )
119
```

Path: `kolibri/core/logger/management/commands/exportlogs.py`
Content:
```
1 import logging
2 import ntpath
3 import os
4 
5 from dateutil import parser
6 from django.conf import settings
7 from django.core.management.base import CommandError
8 from django.utils import translation
9 
10 from kolibri.core.auth.constants.commands_errors import FILE_WRITE_ERROR
11 from kolibri.core.auth.constants.commands_errors import INVALID
12 from kolibri.core.auth.constants.commands_errors import MESSAGES
13 from kolibri.core.auth.constants.commands_errors import NO_FACILITY
14 from kolibri.core.auth.models import Facility
15 from kolibri.core.logger.csv_export import classes_info
16 from kolibri.core.logger.csv_export import csv_file_generator
17 from kolibri.core.logger.models import GenerateCSVLogRequest
18 from kolibri.core.tasks.management.commands.base import AsyncCommand
19 from kolibri.core.tasks.utils import get_current_job
20 from kolibri.utils.time_utils import local_now
21 
22 logger = logging.getLogger(__name__)
23 
24 
25 class Command(AsyncCommand):
26     def add_arguments(self, parser):
27         parser.add_argument(
28             "-O",
29             "--output-file",
30             action="store",
31             dest="output_file",
32             default=None,
33             type=str,
34             help="The generated file will be saved with this name",
35         )
36         parser.add_argument(
37             "-l",
38             "--log-type",
39             action="store",
40             dest="log_type",
41             default="session",
42             choices=classes_info.keys(),
43             help='Log type to be exported. Valid values are "session" and "summary".',
44         )
45         parser.add_argument(
46             "-w",
47             "--overwrite",
48             action="store_true",
49             dest="overwrite",
50             default=False,
51             help="Allows overwritten of the exported file in case it exists",
52         )
53         parser.add_argument(
54             "--facility",
55             action="store",
56             type=str,
57             help="Facility id to import the users into",
58         )
59         parser.add_argument(
60             "--locale",
61             action="store",
62             type=str,
63             default=None,
64             help="Code of the language for the messages to be translated",
65         )
66         parser.add_argument(
67             "--start_date",
68             action="store",
69             dest="start_date",
70             type=str,
71             help="Start date for date range selection of log files. Valid value is an ISO string formatted as YYYY-MM-DDTHH:MM:SS",
72         )
73         parser.add_argument(
74             "--end_date",
75             action="store",
76             dest="end_date",
77             type=str,
78             help="End date for date range selection of log files. Valid value is an ISO string formatted as YYYY-MM-DDTHH:MM:SS",
79         )
80 
81     def get_facility(self, options):
82         if options["facility"]:
83             default_facility = Facility.objects.get(pk=options["facility"])
84         else:
85             default_facility = Facility.get_default_facility()
86 
87         return default_facility
88 
89     def validate_date(self, date_str):
90         try:
91             return bool(parser.parse(date_str))
92         except ValueError:
93             return False
94 
95     def handle_async(self, *args, **options):
96 
97         # set language for the translation of the messages
98         locale = settings.LANGUAGE_CODE if not options["locale"] else options["locale"]
99         translation.activate(locale)
100         self.overall_error = ""
101         job = get_current_job()
102 
103         start_date = options["start_date"]
104         end_date = options["end_date"]
105 
106         facility = self.get_facility(options)
107         if not facility:
108             self.overall_error = str(MESSAGES[NO_FACILITY])
109 
110         elif not self.validate_date(start_date):
111             self.overall_error = str(MESSAGES[INVALID]).format("start_date")
112 
113         elif not self.validate_date(end_date):
114             self.overall_error = str(MESSAGES[INVALID]).format("end_date")
115 
116         else:
117             log_type = options["log_type"]
118 
119             log_info = classes_info[log_type]
120 
121             if options["output_file"] is None:
122                 filename = log_info["filename"].format(
123                     facility.name, facility.id[:4], start_date[:10], end_date[:10]
124                 )
125             else:
126                 filename = options["output_file"]
127 
128             filepath = os.path.join(os.getcwd(), filename)
129 
130             queryset = log_info["queryset"]
131 
132             total_rows = queryset.count()
133 
134             with self.start_progress(total=total_rows) as progress_update:
135                 try:
136                     for row in csv_file_generator(
137                         facility,
138                         log_type,
139                         filepath,
140                         start_date=start_date,
141                         end_date=end_date,
142                         overwrite=options["overwrite"],
143                     ):
144                         progress_update(1)
145                 except (ValueError, IOError) as e:
146                     self.overall_error = str(MESSAGES[FILE_WRITE_ERROR].format(e))
147 
148         if job:
149             job.extra_metadata["overall_error"] = self.overall_error
150             self.job.extra_metadata["filename"] = ntpath.basename(filepath)
151             job.save_meta()
152         else:
153             if self.overall_error:
154                 raise CommandError(self.overall_error)
155             else:
156                 logger.info(
157                     "Created csv file {} with {} lines".format(filepath, total_rows)
158                 )
159 
160         translation.deactivate()
161 
162         # create or update record of log request
163         GenerateCSVLogRequest.objects.update_or_create(
164             log_type=log_type,
165             facility=facility,
166             defaults={
167                 "selected_start_date": parser.parse(start_date),
168                 "selected_end_date": parser.parse(end_date),
169                 "date_requested": local_now(),
170             },
171         )
172
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/kolibri/core/logger/management/commands/exportlogs.py b/kolibri/core/logger/management/commands/exportlogs.py
--- a/kolibri/core/logger/management/commands/exportlogs.py
+++ b/kolibri/core/logger/management/commands/exportlogs.py
@@ -15,6 +15,7 @@
 from kolibri.core.logger.csv_export import classes_info
 from kolibri.core.logger.csv_export import csv_file_generator
 from kolibri.core.logger.models import GenerateCSVLogRequest
+from kolibri.core.logger.tasks import log_exports_cleanup
 from kolibri.core.tasks.management.commands.base import AsyncCommand
 from kolibri.core.tasks.utils import get_current_job
 from kolibri.utils.time_utils import local_now
@@ -169,3 +170,4 @@
                 "date_requested": local_now(),
             },
         )
+        log_exports_cleanup.enqueue()
diff --git a/kolibri/core/logger/tasks.py b/kolibri/core/logger/tasks.py
--- a/kolibri/core/logger/tasks.py
+++ b/kolibri/core/logger/tasks.py
@@ -3,13 +3,19 @@
 from django.core.management import call_command
 from rest_framework import serializers
 
+from kolibri.core.auth.management.commands.bulkexportusers import (
+    CSV_EXPORT_FILENAMES as USER_CSV_EXPORT_FILENAMES,
+)
 from kolibri.core.auth.models import Facility
 from kolibri.core.logger.csv_export import CSV_EXPORT_FILENAMES
+from kolibri.core.logger.models import GenerateCSVLogRequest
 from kolibri.core.tasks.decorators import register_task
 from kolibri.core.tasks.permissions import IsAdminForJob
 from kolibri.core.tasks.validation import JobValidator
 from kolibri.utils import conf
 
+LOGS_CLEANUP_JOB_ID = "18"
+
 
 def get_filepath(log_type, facility_id, start_date, end_date):
     facility = Facility.objects.get(id=facility_id)
@@ -25,6 +31,53 @@
     return filepath
 
 
+def get_valid_logs_csv_filenames():
+    """
+    Returns a set of valid filenames that should exist
+    based on the objects stored in GenerateCSVLogRequest.
+    Any other files except these filenames should be removed.
+    """
+    valid_filenames_set = set()
+    log_requests = GenerateCSVLogRequest.objects.all()
+    for log_request in log_requests:
+        full_path = get_filepath(
+            log_request.log_type,
+            log_request.facility_id,
+            log_request.selected_start_date.strftime("%Y-%m-%d"),
+            log_request.selected_end_date.strftime("%Y-%m-%d"),
+        )
+        valid_filenames_set.add(os.path.basename(full_path))
+    return valid_filenames_set
+
+
+def get_valid_users_csv_filenames():
+    """
+    Returns a set of valid filenames that should exist
+    based on the objects stored in Facility.
+    """
+    valid_filenames_set = set()
+    facilities = Facility.objects.values("id", "name").all()
+    for facility in facilities:
+        file_name = USER_CSV_EXPORT_FILENAMES["user"].format(
+            facility["name"], facility["id"][:4]
+        )
+        valid_filenames_set.add(file_name)
+    return valid_filenames_set
+
+
+def get_valid_filenames():
+    """
+    Returns a union set of valid filenames
+    for log exports and users csv exports.
+    These filenames are valid and will not be
+    cleaned from log_exports_cleanup.
+    """
+    valid_logs_filenames = get_valid_logs_csv_filenames()
+    valid_users_filenames = get_valid_users_csv_filenames()
+    valid_filenames_set = valid_logs_filenames.union(valid_users_filenames)
+    return valid_filenames_set
+
+
 class ExportLogCSVValidator(JobValidator):
     facility = serializers.PrimaryKeyRelatedField(
         queryset=Facility.objects.all(), required=False
@@ -116,3 +169,18 @@
         kwargs.get("end_date"),
         kwargs.get("locale"),
     )
+
+
+@register_task(job_id=LOGS_CLEANUP_JOB_ID)
+def log_exports_cleanup():
+    """
+    Cleanup log_exports csv files that does not have
+    related reocord in GenerateCSVLogRequest model
+    """
+    logs_dir = os.path.join(conf.KOLIBRI_HOME, "log_export")
+    if not os.path.isdir(logs_dir):
+        return
+    valid_filenames_set = get_valid_filenames()
+    for filename in os.listdir(logs_dir):
+        if filename not in valid_filenames_set:
+            os.remove(os.path.join(logs_dir, filename))

{"golden_diff": "diff --git a/kolibri/core/logger/management/commands/exportlogs.py b/kolibri/core/logger/management/commands/exportlogs.py\n--- a/kolibri/core/logger/management/commands/exportlogs.py\n+++ b/kolibri/core/logger/management/commands/exportlogs.py\n@@ -15,6 +15,7 @@\n from kolibri.core.logger.csv_export import classes_info\n from kolibri.core.logger.csv_export import csv_file_generator\n from kolibri.core.logger.models import GenerateCSVLogRequest\n+from kolibri.core.logger.tasks import log_exports_cleanup\n from kolibri.core.tasks.management.commands.base import AsyncCommand\n from kolibri.core.tasks.utils import get_current_job\n from kolibri.utils.time_utils import local_now\n@@ -169,3 +170,4 @@\n                 \"date_requested\": local_now(),\n             },\n         )\n+        log_exports_cleanup.enqueue()\ndiff --git a/kolibri/core/logger/tasks.py b/kolibri/core/logger/tasks.py\n--- a/kolibri/core/logger/tasks.py\n+++ b/kolibri/core/logger/tasks.py\n@@ -3,13 +3,19 @@\n from django.core.management import call_command\n from rest_framework import serializers\n \n+from kolibri.core.auth.management.commands.bulkexportusers import (\n+    CSV_EXPORT_FILENAMES as USER_CSV_EXPORT_FILENAMES,\n+)\n from kolibri.core.auth.models import Facility\n from kolibri.core.logger.csv_export import CSV_EXPORT_FILENAMES\n+from kolibri.core.logger.models import GenerateCSVLogRequest\n from kolibri.core.tasks.decorators import register_task\n from kolibri.core.tasks.permissions import IsAdminForJob\n from kolibri.core.tasks.validation import JobValidator\n from kolibri.utils import conf\n \n+LOGS_CLEANUP_JOB_ID = \"18\"\n+\n \n def get_filepath(log_type, facility_id, start_date, end_date):\n     facility = Facility.objects.get(id=facility_id)\n@@ -25,6 +31,53 @@\n     return filepath\n \n \n+def get_valid_logs_csv_filenames():\n+    \"\"\"\n+    Returns a set of valid filenames that should exist\n+    based on the objects stored in GenerateCSVLogRequest.\n+    Any other files except these filenames should be removed.\n+    \"\"\"\n+    valid_filenames_set = set()\n+    log_requests = GenerateCSVLogRequest.objects.all()\n+    for log_request in log_requests:\n+        full_path = get_filepath(\n+            log_request.log_type,\n+            log_request.facility_id,\n+            log_request.selected_start_date.strftime(\"%Y-%m-%d\"),\n+            log_request.selected_end_date.strftime(\"%Y-%m-%d\"),\n+        )\n+        valid_filenames_set.add(os.path.basename(full_path))\n+    return valid_filenames_set\n+\n+\n+def get_valid_users_csv_filenames():\n+    \"\"\"\n+    Returns a set of valid filenames that should exist\n+    based on the objects stored in Facility.\n+    \"\"\"\n+    valid_filenames_set = set()\n+    facilities = Facility.objects.values(\"id\", \"name\").all()\n+    for facility in facilities:\n+        file_name = USER_CSV_EXPORT_FILENAMES[\"user\"].format(\n+            facility[\"name\"], facility[\"id\"][:4]\n+        )\n+        valid_filenames_set.add(file_name)\n+    return valid_filenames_set\n+\n+\n+def get_valid_filenames():\n+    \"\"\"\n+    Returns a union set of valid filenames\n+    for log exports and users csv exports.\n+    These filenames are valid and will not be\n+    cleaned from log_exports_cleanup.\n+    \"\"\"\n+    valid_logs_filenames = get_valid_logs_csv_filenames()\n+    valid_users_filenames = get_valid_users_csv_filenames()\n+    valid_filenames_set = valid_logs_filenames.union(valid_users_filenames)\n+    return valid_filenames_set\n+\n+\n class ExportLogCSVValidator(JobValidator):\n     facility = serializers.PrimaryKeyRelatedField(\n         queryset=Facility.objects.all(), required=False\n@@ -116,3 +169,18 @@\n         kwargs.get(\"end_date\"),\n         kwargs.get(\"locale\"),\n     )\n+\n+\n+@register_task(job_id=LOGS_CLEANUP_JOB_ID)\n+def log_exports_cleanup():\n+    \"\"\"\n+    Cleanup log_exports csv files that does not have\n+    related reocord in GenerateCSVLogRequest model\n+    \"\"\"\n+    logs_dir = os.path.join(conf.KOLIBRI_HOME, \"log_export\")\n+    if not os.path.isdir(logs_dir):\n+        return\n+    valid_filenames_set = get_valid_filenames()\n+    for filename in os.listdir(logs_dir):\n+        if filename not in valid_filenames_set:\n+            os.remove(os.path.join(logs_dir, filename))\n", "issue": "Clean up previously generated CSV files\n<!--\r\nInstructions:\r\n * Fill out the sections below, replace \u2026's with information about your issue\r\n * Use the 'preview' function above this text box to verify formatting before submitting\r\n-->\r\n\r\n## Observed behavior\r\nWhen a new CSV file is generated for session and summary logs, the previously generated files that are stored in the user's database are no longer accessible for download in the UI. A record of the new CSV file request is saved to the logger model `GenerateCSVLogRequest` which contains the facility id, log type (session or summary), the selected start date, the selected end date, and the date that the request was made. There should only ever be one record for each facility and log type combination. The information in this modal is used to scan the database folder `log_exports` and retrieve the previously generated file for download.\r\n\r\nNote: For session and summary log files, the file naming convention has been updated to include a date-range selection, with the dates formatted as `YYYY-MM-DD`.\r\nPrevious naming convention:\r\n- `{facility_name}_{last 4 digits of facility ID}_{log export type}.csv`\r\n\r\nCurrent naming convention:\r\n- `{facility_name}_{last 4 digits of facility ID}_{log export type}_from_{start date}_to_{end date}.csv`\r\n\r\n## Expected behavior\r\nWe should remove any files that do not have a record in the database. One way to achieve this is by scanning the folder `log_export`, which contains all exported log files, using the predictable file naming convention, and cross-referencing them with the file records stored in the database. Any session or summary files that do not have a corresponding record in the database should be deleted.\r\n\r\nWith a 1:1 correlation with the `GenerateCSVLogRequest` record, the database should only contain one exported file for each session and summary log type. A new async task should be created for performing this cleanup, and can be queued whenever a new log export is generated. If the task has a predictable id, it can be ensured that there is only one cleanup task going at once.\r\n\r\nMore details on the task system can be found [here](https://kolibri-dev.readthedocs.io/en/develop/backend_architecture/tasks.html).\r\n\r\n## User-facing consequences\r\nEnsures the database is free from redundant files and optimizes the user's database storage.\r\n\r\n## Acceptance Criteria\r\n\r\n- [ ] All previously generated session and summary log file exports that lack a corresponding record in the database have been deleted.\r\n- [ ] The previously generated CSV file should be automatically removed from the database upon generating a new one.\r\n- [ ] New python code should be covered by unit tests.\r\n\n", "before_files": [{"content": "import os\n\nfrom django.core.management import call_command\nfrom rest_framework import serializers\n\nfrom kolibri.core.auth.models import Facility\nfrom kolibri.core.logger.csv_export import CSV_EXPORT_FILENAMES\nfrom kolibri.core.tasks.decorators import register_task\nfrom kolibri.core.tasks.permissions import IsAdminForJob\nfrom kolibri.core.tasks.validation import JobValidator\nfrom kolibri.utils import conf\n\n\ndef get_filepath(log_type, facility_id, start_date, end_date):\n    facility = Facility.objects.get(id=facility_id)\n    logs_dir = os.path.join(conf.KOLIBRI_HOME, \"log_export\")\n    if not os.path.isdir(logs_dir):\n        os.mkdir(logs_dir)\n    filepath = os.path.join(\n        logs_dir,\n        CSV_EXPORT_FILENAMES[log_type].format(\n            facility.name, facility.id[:4], start_date[:10], end_date[:10]\n        ),\n    )\n    return filepath\n\n\nclass ExportLogCSVValidator(JobValidator):\n    facility = serializers.PrimaryKeyRelatedField(\n        queryset=Facility.objects.all(), required=False\n    )\n    start_date = serializers.CharField()\n    end_date = serializers.CharField()\n    locale = serializers.CharField(required=False)\n\n    def validate(self, data):\n        facility = data.get(\"facility\", None)\n        start_date = data.get(\"start_date\", None)\n        end_date = data.get(\"end_date\", None)\n        locale = data.get(\"locale\", None)\n\n        if facility is None and \"user\" in self.context:\n            facility = self.context[\"user\"].facility\n        elif facility is None:\n            raise serializers.ValidationError(\n                \"Facility must be specified when no user is available.\"\n            )\n        if not start_date or not end_date:\n            raise serializers.ValidationError(\n                \"Start {} and End {} date values are required.\".format(\n                    start_date, end_date\n                )\n            )\n        kwargs = {\n            \"facility\": facility.id,\n            \"start_date\": start_date,\n            \"end_date\": end_date,\n            \"locale\": locale,\n        }\n        return {\n            \"facility_id\": facility.id,\n            \"kwargs\": kwargs,\n            \"args\": [facility.id],\n        }\n\n\ndef _exportlogcsv(log_type, facility_id, start_date, end_date, locale):\n    filepath = get_filepath(log_type, facility_id, start_date, end_date)\n    call_command(\n        \"exportlogs\",\n        log_type=log_type,\n        output_file=filepath,\n        facility=facility_id,\n        overwrite=True,\n        start_date=start_date,\n        end_date=end_date,\n        locale=locale,\n    )\n\n\n@register_task(\n    validator=ExportLogCSVValidator,\n    track_progress=True,\n    permission_classes=[IsAdminForJob],\n)\ndef exportsessionlogcsv(facility_id, **kwargs):\n    \"\"\"\n    Dumps in csv format the content session logs.\n\n    :param: facility.\n    \"\"\"\n    _exportlogcsv(\n        \"session\",\n        facility_id,\n        kwargs.get(\"start_date\"),\n        kwargs.get(\"end_date\"),\n        kwargs.get(\"locale\"),\n    )\n\n\n@register_task(\n    validator=ExportLogCSVValidator,\n    track_progress=True,\n    permission_classes=[IsAdminForJob],\n)\ndef exportsummarylogcsv(facility_id, **kwargs):\n    \"\"\"\n    Dumps in csv format the content summary logs.\n\n    :param: facility.\n    \"\"\"\n    _exportlogcsv(\n        \"summary\",\n        facility_id,\n        kwargs.get(\"start_date\"),\n        kwargs.get(\"end_date\"),\n        kwargs.get(\"locale\"),\n    )\n", "path": "kolibri/core/logger/tasks.py"}, {"content": "import logging\nimport ntpath\nimport os\n\nfrom dateutil import parser\nfrom django.conf import settings\nfrom django.core.management.base import CommandError\nfrom django.utils import translation\n\nfrom kolibri.core.auth.constants.commands_errors import FILE_WRITE_ERROR\nfrom kolibri.core.auth.constants.commands_errors import INVALID\nfrom kolibri.core.auth.constants.commands_errors import MESSAGES\nfrom kolibri.core.auth.constants.commands_errors import NO_FACILITY\nfrom kolibri.core.auth.models import Facility\nfrom kolibri.core.logger.csv_export import classes_info\nfrom kolibri.core.logger.csv_export import csv_file_generator\nfrom kolibri.core.logger.models import GenerateCSVLogRequest\nfrom kolibri.core.tasks.management.commands.base import AsyncCommand\nfrom kolibri.core.tasks.utils import get_current_job\nfrom kolibri.utils.time_utils import local_now\n\nlogger = logging.getLogger(__name__)\n\n\nclass Command(AsyncCommand):\n    def add_arguments(self, parser):\n        parser.add_argument(\n            \"-O\",\n            \"--output-file\",\n            action=\"store\",\n            dest=\"output_file\",\n            default=None,\n            type=str,\n            help=\"The generated file will be saved with this name\",\n        )\n        parser.add_argument(\n            \"-l\",\n            \"--log-type\",\n            action=\"store\",\n            dest=\"log_type\",\n            default=\"session\",\n            choices=classes_info.keys(),\n            help='Log type to be exported. Valid values are \"session\" and \"summary\".',\n        )\n        parser.add_argument(\n            \"-w\",\n            \"--overwrite\",\n            action=\"store_true\",\n            dest=\"overwrite\",\n            default=False,\n            help=\"Allows overwritten of the exported file in case it exists\",\n        )\n        parser.add_argument(\n            \"--facility\",\n            action=\"store\",\n            type=str,\n            help=\"Facility id to import the users into\",\n        )\n        parser.add_argument(\n            \"--locale\",\n            action=\"store\",\n            type=str,\n            default=None,\n            help=\"Code of the language for the messages to be translated\",\n        )\n        parser.add_argument(\n            \"--start_date\",\n            action=\"store\",\n            dest=\"start_date\",\n            type=str,\n            help=\"Start date for date range selection of log files. Valid value is an ISO string formatted as YYYY-MM-DDTHH:MM:SS\",\n        )\n        parser.add_argument(\n            \"--end_date\",\n            action=\"store\",\n            dest=\"end_date\",\n            type=str,\n            help=\"End date for date range selection of log files. Valid value is an ISO string formatted as YYYY-MM-DDTHH:MM:SS\",\n        )\n\n    def get_facility(self, options):\n        if options[\"facility\"]:\n            default_facility = Facility.objects.get(pk=options[\"facility\"])\n        else:\n            default_facility = Facility.get_default_facility()\n\n        return default_facility\n\n    def validate_date(self, date_str):\n        try:\n            return bool(parser.parse(date_str))\n        except ValueError:\n            return False\n\n    def handle_async(self, *args, **options):\n\n        # set language for the translation of the messages\n        locale = settings.LANGUAGE_CODE if not options[\"locale\"] else options[\"locale\"]\n        translation.activate(locale)\n        self.overall_error = \"\"\n        job = get_current_job()\n\n        start_date = options[\"start_date\"]\n        end_date = options[\"end_date\"]\n\n        facility = self.get_facility(options)\n        if not facility:\n            self.overall_error = str(MESSAGES[NO_FACILITY])\n\n        elif not self.validate_date(start_date):\n            self.overall_error = str(MESSAGES[INVALID]).format(\"start_date\")\n\n        elif not self.validate_date(end_date):\n            self.overall_error = str(MESSAGES[INVALID]).format(\"end_date\")\n\n        else:\n            log_type = options[\"log_type\"]\n\n            log_info = classes_info[log_type]\n\n            if options[\"output_file\"] is None:\n                filename = log_info[\"filename\"].format(\n                    facility.name, facility.id[:4], start_date[:10], end_date[:10]\n                )\n            else:\n                filename = options[\"output_file\"]\n\n            filepath = os.path.join(os.getcwd(), filename)\n\n            queryset = log_info[\"queryset\"]\n\n            total_rows = queryset.count()\n\n            with self.start_progress(total=total_rows) as progress_update:\n                try:\n                    for row in csv_file_generator(\n                        facility,\n                        log_type,\n                        filepath,\n                        start_date=start_date,\n                        end_date=end_date,\n                        overwrite=options[\"overwrite\"],\n                    ):\n                        progress_update(1)\n                except (ValueError, IOError) as e:\n                    self.overall_error = str(MESSAGES[FILE_WRITE_ERROR].format(e))\n\n        if job:\n            job.extra_metadata[\"overall_error\"] = self.overall_error\n            self.job.extra_metadata[\"filename\"] = ntpath.basename(filepath)\n            job.save_meta()\n        else:\n            if self.overall_error:\n                raise CommandError(self.overall_error)\n            else:\n                logger.info(\n                    \"Created csv file {} with {} lines\".format(filepath, total_rows)\n                )\n\n        translation.deactivate()\n\n        # create or update record of log request\n        GenerateCSVLogRequest.objects.update_or_create(\n            log_type=log_type,\n            facility=facility,\n            defaults={\n                \"selected_start_date\": parser.parse(start_date),\n                \"selected_end_date\": parser.parse(end_date),\n                \"date_requested\": local_now(),\n            },\n        )\n", "path": "kolibri/core/logger/management/commands/exportlogs.py"}], "after_files": [{"content": "import os\n\nfrom django.core.management import call_command\nfrom rest_framework import serializers\n\nfrom kolibri.core.auth.management.commands.bulkexportusers import (\n    CSV_EXPORT_FILENAMES as USER_CSV_EXPORT_FILENAMES,\n)\nfrom kolibri.core.auth.models import Facility\nfrom kolibri.core.logger.csv_export import CSV_EXPORT_FILENAMES\nfrom kolibri.core.logger.models import GenerateCSVLogRequest\nfrom kolibri.core.tasks.decorators import register_task\nfrom kolibri.core.tasks.permissions import IsAdminForJob\nfrom kolibri.core.tasks.validation import JobValidator\nfrom kolibri.utils import conf\n\nLOGS_CLEANUP_JOB_ID = \"18\"\n\n\ndef get_filepath(log_type, facility_id, start_date, end_date):\n    facility = Facility.objects.get(id=facility_id)\n    logs_dir = os.path.join(conf.KOLIBRI_HOME, \"log_export\")\n    if not os.path.isdir(logs_dir):\n        os.mkdir(logs_dir)\n    filepath = os.path.join(\n        logs_dir,\n        CSV_EXPORT_FILENAMES[log_type].format(\n            facility.name, facility.id[:4], start_date[:10], end_date[:10]\n        ),\n    )\n    return filepath\n\n\ndef get_valid_logs_csv_filenames():\n    \"\"\"\n    Returns a set of valid filenames that should exist\n    based on the objects stored in GenerateCSVLogRequest.\n    Any other files except these filenames should be removed.\n    \"\"\"\n    valid_filenames_set = set()\n    log_requests = GenerateCSVLogRequest.objects.all()\n    for log_request in log_requests:\n        full_path = get_filepath(\n            log_request.log_type,\n            log_request.facility_id,\n            log_request.selected_start_date.strftime(\"%Y-%m-%d\"),\n            log_request.selected_end_date.strftime(\"%Y-%m-%d\"),\n        )\n        valid_filenames_set.add(os.path.basename(full_path))\n    return valid_filenames_set\n\n\ndef get_valid_users_csv_filenames():\n    \"\"\"\n    Returns a set of valid filenames that should exist\n    based on the objects stored in Facility.\n    \"\"\"\n    valid_filenames_set = set()\n    facilities = Facility.objects.values(\"id\", \"name\").all()\n    for facility in facilities:\n        file_name = USER_CSV_EXPORT_FILENAMES[\"user\"].format(\n            facility[\"name\"], facility[\"id\"][:4]\n        )\n        valid_filenames_set.add(file_name)\n    return valid_filenames_set\n\n\ndef get_valid_filenames():\n    \"\"\"\n    Returns a union set of valid filenames\n    for log exports and users csv exports.\n    These filenames are valid and will not be\n    cleaned from log_exports_cleanup.\n    \"\"\"\n    valid_logs_filenames = get_valid_logs_csv_filenames()\n    valid_users_filenames = get_valid_users_csv_filenames()\n    valid_filenames_set = valid_logs_filenames.union(valid_users_filenames)\n    return valid_filenames_set\n\n\nclass ExportLogCSVValidator(JobValidator):\n    facility = serializers.PrimaryKeyRelatedField(\n        queryset=Facility.objects.all(), required=False\n    )\n    start_date = serializers.CharField()\n    end_date = serializers.CharField()\n    locale = serializers.CharField(required=False)\n\n    def validate(self, data):\n        facility = data.get(\"facility\", None)\n        start_date = data.get(\"start_date\", None)\n        end_date = data.get(\"end_date\", None)\n        locale = data.get(\"locale\", None)\n\n        if facility is None and \"user\" in self.context:\n            facility = self.context[\"user\"].facility\n        elif facility is None:\n            raise serializers.ValidationError(\n                \"Facility must be specified when no user is available.\"\n            )\n        if not start_date or not end_date:\n            raise serializers.ValidationError(\n                \"Start {} and End {} date values are required.\".format(\n                    start_date, end_date\n                )\n            )\n        kwargs = {\n            \"facility\": facility.id,\n            \"start_date\": start_date,\n            \"end_date\": end_date,\n            \"locale\": locale,\n        }\n        return {\n            \"facility_id\": facility.id,\n            \"kwargs\": kwargs,\n            \"args\": [facility.id],\n        }\n\n\ndef _exportlogcsv(log_type, facility_id, start_date, end_date, locale):\n    filepath = get_filepath(log_type, facility_id, start_date, end_date)\n    call_command(\n        \"exportlogs\",\n        log_type=log_type,\n        output_file=filepath,\n        facility=facility_id,\n        overwrite=True,\n        start_date=start_date,\n        end_date=end_date,\n        locale=locale,\n    )\n\n\n@register_task(\n    validator=ExportLogCSVValidator,\n    track_progress=True,\n    permission_classes=[IsAdminForJob],\n)\ndef exportsessionlogcsv(facility_id, **kwargs):\n    \"\"\"\n    Dumps in csv format the content session logs.\n\n    :param: facility.\n    \"\"\"\n    _exportlogcsv(\n        \"session\",\n        facility_id,\n        kwargs.get(\"start_date\"),\n        kwargs.get(\"end_date\"),\n        kwargs.get(\"locale\"),\n    )\n\n\n@register_task(\n    validator=ExportLogCSVValidator,\n    track_progress=True,\n    permission_classes=[IsAdminForJob],\n)\ndef exportsummarylogcsv(facility_id, **kwargs):\n    \"\"\"\n    Dumps in csv format the content summary logs.\n\n    :param: facility.\n    \"\"\"\n    _exportlogcsv(\n        \"summary\",\n        facility_id,\n        kwargs.get(\"start_date\"),\n        kwargs.get(\"end_date\"),\n        kwargs.get(\"locale\"),\n    )\n\n\n@register_task(job_id=LOGS_CLEANUP_JOB_ID)\ndef log_exports_cleanup():\n    \"\"\"\n    Cleanup log_exports csv files that does not have\n    related reocord in GenerateCSVLogRequest model\n    \"\"\"\n    logs_dir = os.path.join(conf.KOLIBRI_HOME, \"log_export\")\n    if not os.path.isdir(logs_dir):\n        return\n    valid_filenames_set = get_valid_filenames()\n    for filename in os.listdir(logs_dir):\n        if filename not in valid_filenames_set:\n            os.remove(os.path.join(logs_dir, filename))\n", "path": "kolibri/core/logger/tasks.py"}, {"content": "import logging\nimport ntpath\nimport os\n\nfrom dateutil import parser\nfrom django.conf import settings\nfrom django.core.management.base import CommandError\nfrom django.utils import translation\n\nfrom kolibri.core.auth.constants.commands_errors import FILE_WRITE_ERROR\nfrom kolibri.core.auth.constants.commands_errors import INVALID\nfrom kolibri.core.auth.constants.commands_errors import MESSAGES\nfrom kolibri.core.auth.constants.commands_errors import NO_FACILITY\nfrom kolibri.core.auth.models import Facility\nfrom kolibri.core.logger.csv_export import classes_info\nfrom kolibri.core.logger.csv_export import csv_file_generator\nfrom kolibri.core.logger.models import GenerateCSVLogRequest\nfrom kolibri.core.logger.tasks import log_exports_cleanup\nfrom kolibri.core.tasks.management.commands.base import AsyncCommand\nfrom kolibri.core.tasks.utils import get_current_job\nfrom kolibri.utils.time_utils import local_now\n\nlogger = logging.getLogger(__name__)\n\n\nclass Command(AsyncCommand):\n    def add_arguments(self, parser):\n        parser.add_argument(\n            \"-O\",\n            \"--output-file\",\n            action=\"store\",\n            dest=\"output_file\",\n            default=None,\n            type=str,\n            help=\"The generated file will be saved with this name\",\n        )\n        parser.add_argument(\n            \"-l\",\n            \"--log-type\",\n            action=\"store\",\n            dest=\"log_type\",\n            default=\"session\",\n            choices=classes_info.keys(),\n            help='Log type to be exported. Valid values are \"session\" and \"summary\".',\n        )\n        parser.add_argument(\n            \"-w\",\n            \"--overwrite\",\n            action=\"store_true\",\n            dest=\"overwrite\",\n            default=False,\n            help=\"Allows overwritten of the exported file in case it exists\",\n        )\n        parser.add_argument(\n            \"--facility\",\n            action=\"store\",\n            type=str,\n            help=\"Facility id to import the users into\",\n        )\n        parser.add_argument(\n            \"--locale\",\n            action=\"store\",\n            type=str,\n            default=None,\n            help=\"Code of the language for the messages to be translated\",\n        )\n        parser.add_argument(\n            \"--start_date\",\n            action=\"store\",\n            dest=\"start_date\",\n            type=str,\n            help=\"Start date for date range selection of log files. Valid value is an ISO string formatted as YYYY-MM-DDTHH:MM:SS\",\n        )\n        parser.add_argument(\n            \"--end_date\",\n            action=\"store\",\n            dest=\"end_date\",\n            type=str,\n            help=\"End date for date range selection of log files. Valid value is an ISO string formatted as YYYY-MM-DDTHH:MM:SS\",\n        )\n\n    def get_facility(self, options):\n        if options[\"facility\"]:\n            default_facility = Facility.objects.get(pk=options[\"facility\"])\n        else:\n            default_facility = Facility.get_default_facility()\n\n        return default_facility\n\n    def validate_date(self, date_str):\n        try:\n            return bool(parser.parse(date_str))\n        except ValueError:\n            return False\n\n    def handle_async(self, *args, **options):\n\n        # set language for the translation of the messages\n        locale = settings.LANGUAGE_CODE if not options[\"locale\"] else options[\"locale\"]\n        translation.activate(locale)\n        self.overall_error = \"\"\n        job = get_current_job()\n\n        start_date = options[\"start_date\"]\n        end_date = options[\"end_date\"]\n\n        facility = self.get_facility(options)\n        if not facility:\n            self.overall_error = str(MESSAGES[NO_FACILITY])\n\n        elif not self.validate_date(start_date):\n            self.overall_error = str(MESSAGES[INVALID]).format(\"start_date\")\n\n        elif not self.validate_date(end_date):\n            self.overall_error = str(MESSAGES[INVALID]).format(\"end_date\")\n\n        else:\n            log_type = options[\"log_type\"]\n\n            log_info = classes_info[log_type]\n\n            if options[\"output_file\"] is None:\n                filename = log_info[\"filename\"].format(\n                    facility.name, facility.id[:4], start_date[:10], end_date[:10]\n                )\n            else:\n                filename = options[\"output_file\"]\n\n            filepath = os.path.join(os.getcwd(), filename)\n\n            queryset = log_info[\"queryset\"]\n\n            total_rows = queryset.count()\n\n            with self.start_progress(total=total_rows) as progress_update:\n                try:\n                    for row in csv_file_generator(\n                        facility,\n                        log_type,\n                        filepath,\n                        start_date=start_date,\n                        end_date=end_date,\n                        overwrite=options[\"overwrite\"],\n                    ):\n                        progress_update(1)\n                except (ValueError, IOError) as e:\n                    self.overall_error = str(MESSAGES[FILE_WRITE_ERROR].format(e))\n\n        if job:\n            job.extra_metadata[\"overall_error\"] = self.overall_error\n            self.job.extra_metadata[\"filename\"] = ntpath.basename(filepath)\n            job.save_meta()\n        else:\n            if self.overall_error:\n                raise CommandError(self.overall_error)\n            else:\n                logger.info(\n                    \"Created csv file {} with {} lines\".format(filepath, total_rows)\n                )\n\n        translation.deactivate()\n\n        # create or update record of log request\n        GenerateCSVLogRequest.objects.update_or_create(\n            log_type=log_type,\n            facility=facility,\n            defaults={\n                \"selected_start_date\": parser.parse(start_date),\n                \"selected_end_date\": parser.parse(end_date),\n                \"date_requested\": local_now(),\n            },\n        )\n        log_exports_cleanup.enqueue()\n", "path": "kolibri/core/logger/management/commands/exportlogs.py"}]}