prompt: We are currently solving the following issue within our repository. Here is the issue text:
  --- BEGIN ISSUE ---
  Overriding rspamd forbidden_file_extension.map doesn't work as documented
  <!--
  
  Thank you for opening an issue with Mailu. Please understand that issues are meant for bugs and enhancement-requests.
  For **user-support questions**, reach out to us  on [matrix](https://matrix.to/#/#mailu:tedomum.net).
  
  To be able to help you best, we need some more information.
  
  Before you open your issue
  - Check if no issue or pull-request for this already exists.
  - Check [documentation](https://mailu.io/master/) and [FAQ](https://mailu.io/master/faq.html). (Tip, use the search function on the documentation page)
  - You understand `Mailu` is made by volunteers in their **free time** — be concise, civil and accept that delays can occur.
  - The title of the issue should be short and simple. It should contain specific terms related to the actual issue. Be specific while writing the title.
  
  Please put your text outside of the comment blocks to be visible. You can use the button "Preview" above to check.
  
  -->
  
  ## Environment & Version
  The host is Ubuntu 22.04
  ### Environment
  
  - [x] docker compose
  - [ ] kubernetes
  - [ ] docker swarm
  
  ### Version
  
  - Version: `2.0`
  
  <!--
  To find your version, get the image name of a mailu container and read  the version from the tag (example for version 1.7).
  
  $> docker ps -a | grep mailu
  140b09d4b09c    mailu/roundcube:1.7    "docker-php-entrypoi…"    2 weeks ago    Up 2 days (healthy)    80/tcp
  $> grep MAILU_VERSION docker-compose.yml mailu.env
  -->
  
  ## Description
  <!--
  Further explain the bug in a few words. It should be clear what the unexpected behaviour is.  Share it in an easy-to-understand language.
  -->
  My forbidden_file_extension.map placed in `overrides/rspamd/` is ignored and the default file is used instead.
  
  ## Replication Steps
  <!--
  Steps for replicating your issue
  -->
  
  With a Mailu 2.0 instance, follow the [instructions](https://mailu.io/2.0/antispam.html#can-i-change-the-list-of-authorized-file-attachments) for changing allowed attachments.  Change `overrides/rspamd/forbidden_file_extension.map` and add/remove some lines, then run `docker compose restart antispam`.  Re-run the `docker compose exec antispam cat /etc/rspamd/local.d/forbidden_file_extension.map` command to see the file used by rspamd.
  
  ## Observed behaviour
  <!--
  Explain or paste the result you received.
  -->
  I get the output of `core/rspamd/conf/forbidden_file_extension.map` from the Mailu distribution.  Indeed, attachments allowed are still in line with this file, rather than the new changed file.
  
  ## Expected behaviour
  <!--
  Explain what results you expected - be as specific as possible.
  Just saying "it doesn’t work as expected" is not useful. It's also helpful to describe what you actually experienced.
  -->
  I'd expect the modified forbidden_file_extension.map in the `overrides/rspamd` directory is taken into account.  For instance, if I remove js from the list, I should be able to send an attachment with the js extension.
  
  <!--
  Often it is very useful to include log fragments of the involved component.
  You can get the logs via `docker logs <container name> --tail 1000`.
  For example for the admin container: `docker logs mailu_admin_1 --tail 1000`
  or using docker compose `docker compose -f /mailu/docker-compose.yml logs --tail 1000 admin`
  
  If you can find the relevant section, please share only the parts that seem relevant. If you have any logs, please enclose them in code tags, like so:
  
  ```
  Your logs here!
  ```
  -->
  ## My diagnosis
  The overridden file is available at `/overrides/forbidden_file_extension.map` in the antispam container, as expected.  However, `core/rspamd/start.py` adds `forbidden_file_extension.map` to the `config_files` variable so the override file at `/overrides/forbidden_file_extension.map` is ignored and not copied over.  The start script runs Jinja on `/conf/forbidden_file_extension.map` but since there's no directives it's just copied verbatim to `/etc/rspamd/local.d/`.  This default file is what rspamd reads.  The `multimap.conf` file references the map file, but that has no directive for overriding the path of the map file so overriding `multimap.conf` doesn't help.
  
  ## Possible solution
  To get the expected behaviour (my override file is copied to `/etc/rspamd/local.d/` instead of the default file), I suspect the start script needs to be altered.  I'm unsure how this was intended to be done originally, but I'd probably modify the start script so an administrator can specify whether certain override configuration files should be copied despite being on the `config_files` list.  A pleasant way to still keep Jinja flexibility is for the copied override configuration files to be processed as templates too.  For instance I could copy the `multimap.conf` template to a file in the overrides directory, change it and let the start script process the new template.  This is nicely used in Kolla Ansible for instance.  If it's something relatively straightforward like this I'd be happy to make a PR for it, but might take some time since it's my first issue/PR.
  
  --- END ISSUE ---
  
  Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
  --- BEGIN FILES ---
  Path: `core/rspamd/start.py`
  Content:
  ```
  1 #!/usr/bin/env python3
  2 
  3 import os
  4 import glob
  5 import logging as log
  6 import requests
  7 import shutil
  8 import sys
  9 import time
  10 from socrate import system,conf
  11 
  12 env = system.set_env()
  13 
  14 # Actual startup script
  15 
  16 config_files = []
  17 for rspamd_file in glob.glob("/conf/*"):
  18     conf.jinja(rspamd_file, env, os.path.join("/etc/rspamd/local.d", os.path.basename(rspamd_file)))
  19     config_files.append(os.path.basename(rspamd_file))
  20 
  21 for override_file in glob.glob("/overrides/*"):
  22     if os.path.basename(override_file) not in config_files:
  23         shutil.copyfile(override_file, os.path.join("/etc/rspamd/local.d", os.path.basename(override_file)))
  24 
  25 # Admin may not be up just yet
  26 healthcheck = f'http://{env["ADMIN_ADDRESS"]}:8080/internal/rspamd/local_domains'
  27 while True:
  28     time.sleep(1)
  29     try:
  30         if requests.get(healthcheck,timeout=2).ok:
  31             break
  32     except:
  33         pass
  34     log.warning("Admin is not up just yet, retrying in 1 second")
  35 
  36 # Run rspamd
  37 os.system("mkdir -m 755 -p /run/rspamd")
  38 os.system("chown rspamd:rspamd /run/rspamd")
  39 os.system("find /var/lib/rspamd | grep -v /filter | xargs -n1 chown rspamd:rspamd")
  40 os.execv("/usr/bin/rspamd", ["rspamd", "-f", "-u", "rspamd", "-g", "rspamd"])
  41
  ```
  --- END FILES ---
  
  Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.
  
  Here is an example:
  
  ```diff
  diff --git a/examples/server_async.py b/examples/server_async.py
  --- a/examples/server_async.py
  +++ b/examples/server_async.py
  @@ -313,4 +313,4 @@
   
   
   if __name__ == "__main__":
  -    asyncio.run(run_async_server("."), debug=True)
  +    asyncio.run(run_async_server(), debug=True)
  diff --git a/examples/server_sync.py b/examples/server_sync.py
  --- a/examples/server_sync.py
  +++ b/examples/server_sync.py
  @@ -313,5 +313,5 @@
   
   
   if __name__ == "__main__":
  -    server = run_sync_server(".")
  +    server = run_sync_server()
       server.shutdown()
  
  ```
  
golden_diff: diff --git a/core/rspamd/start.py b/core/rspamd/start.py
  --- a/core/rspamd/start.py
  +++ b/core/rspamd/start.py
  @@ -16,7 +16,8 @@
   config_files = []
   for rspamd_file in glob.glob("/conf/*"):
       conf.jinja(rspamd_file, env, os.path.join("/etc/rspamd/local.d", os.path.basename(rspamd_file)))
  -    config_files.append(os.path.basename(rspamd_file))
  +    if rspamd_file != '/conf/forbidden_file_extension.map':
  +        config_files.append(os.path.basename(rspamd_file))
   
   for override_file in glob.glob("/overrides/*"):
       if os.path.basename(override_file) not in config_files:
  
verification_info: {"golden_diff": "diff --git a/core/rspamd/start.py b/core/rspamd/start.py\n--- a/core/rspamd/start.py\n+++ b/core/rspamd/start.py\n@@ -16,7 +16,8 @@\n config_files = []\n for rspamd_file in glob.glob(\"/conf/*\"):\n     conf.jinja(rspamd_file, env, os.path.join(\"/etc/rspamd/local.d\", os.path.basename(rspamd_file)))\n-    config_files.append(os.path.basename(rspamd_file))\n+    if rspamd_file != '/conf/forbidden_file_extension.map':\n+        config_files.append(os.path.basename(rspamd_file))\n \n for override_file in glob.glob(\"/overrides/*\"):\n     if os.path.basename(override_file) not in config_files:\n", "issue": "Overriding rspamd forbidden_file_extension.map doesn't work as documented\n<!--\r\n\r\nThank you for opening an issue with Mailu. Please understand that issues are meant for bugs and enhancement-requests.\r\nFor **user-support questions**, reach out to us  on [matrix](https://matrix.to/#/#mailu:tedomum.net).\r\n\r\nTo be able to help you best, we need some more information.\r\n\r\nBefore you open your issue\r\n- Check if no issue or pull-request for this already exists.\r\n- Check [documentation](https://mailu.io/master/) and [FAQ](https://mailu.io/master/faq.html). (Tip, use the search function on the documentation page)\r\n- You understand `Mailu` is made by volunteers in their **free time** \u2014 be concise, civil and accept that delays can occur.\r\n- The title of the issue should be short and simple. It should contain specific terms related to the actual issue. Be specific while writing the title.\r\n\r\nPlease put your text outside of the comment blocks to be visible. You can use the button \"Preview\" above to check.\r\n\r\n-->\r\n\r\n## Environment & Version\r\nThe host is Ubuntu 22.04\r\n### Environment\r\n\r\n- [x] docker compose\r\n- [ ] kubernetes\r\n- [ ] docker swarm\r\n\r\n### Version\r\n\r\n- Version: `2.0`\r\n\r\n<!--\r\nTo find your version, get the image name of a mailu container and read  the version from the tag (example for version 1.7).\r\n\r\n$> docker ps -a | grep mailu\r\n140b09d4b09c    mailu/roundcube:1.7    \"docker-php-entrypoi\u2026\"    2 weeks ago    Up 2 days (healthy)    80/tcp\r\n$> grep MAILU_VERSION docker-compose.yml mailu.env\r\n-->\r\n\r\n## Description\r\n<!--\r\nFurther explain the bug in a few words. It should be clear what the unexpected behaviour is.  Share it in an easy-to-understand language.\r\n-->\r\nMy forbidden_file_extension.map placed in `overrides/rspamd/` is ignored and the default file is used instead.\r\n\r\n## Replication Steps\r\n<!--\r\nSteps for replicating your issue\r\n-->\r\n\r\nWith a Mailu 2.0 instance, follow the [instructions](https://mailu.io/2.0/antispam.html#can-i-change-the-list-of-authorized-file-attachments) for changing allowed attachments.  Change `overrides/rspamd/forbidden_file_extension.map` and add/remove some lines, then run `docker compose restart antispam`.  Re-run the `docker compose exec antispam cat /etc/rspamd/local.d/forbidden_file_extension.map` command to see the file used by rspamd.\r\n\r\n## Observed behaviour\r\n<!--\r\nExplain or paste the result you received.\r\n-->\r\nI get the output of `core/rspamd/conf/forbidden_file_extension.map` from the Mailu distribution.  Indeed, attachments allowed are still in line with this file, rather than the new changed file.\r\n\r\n## Expected behaviour\r\n<!--\r\nExplain what results you expected - be as specific as possible.\r\nJust saying \"it doesn\u2019t work as expected\" is not useful. It's also helpful to describe what you actually experienced.\r\n-->\r\nI'd expect the modified forbidden_file_extension.map in the `overrides/rspamd` directory is taken into account.  For instance, if I remove js from the list, I should be able to send an attachment with the js extension.\r\n\r\n<!--\r\nOften it is very useful to include log fragments of the involved component.\r\nYou can get the logs via `docker logs <container name> --tail 1000`.\r\nFor example for the admin container: `docker logs mailu_admin_1 --tail 1000`\r\nor using docker compose `docker compose -f /mailu/docker-compose.yml logs --tail 1000 admin`\r\n\r\nIf you can find the relevant section, please share only the parts that seem relevant. If you have any logs, please enclose them in code tags, like so:\r\n\r\n```\r\nYour logs here!\r\n```\r\n-->\r\n## My diagnosis\r\nThe overridden file is available at `/overrides/forbidden_file_extension.map` in the antispam container, as expected.  However, `core/rspamd/start.py` adds `forbidden_file_extension.map` to the `config_files` variable so the override file at `/overrides/forbidden_file_extension.map` is ignored and not copied over.  The start script runs Jinja on `/conf/forbidden_file_extension.map` but since there's no directives it's just copied verbatim to `/etc/rspamd/local.d/`.  This default file is what rspamd reads.  The `multimap.conf` file references the map file, but that has no directive for overriding the path of the map file so overriding `multimap.conf` doesn't help.\r\n\r\n## Possible solution\r\nTo get the expected behaviour (my override file is copied to `/etc/rspamd/local.d/` instead of the default file), I suspect the start script needs to be altered.  I'm unsure how this was intended to be done originally, but I'd probably modify the start script so an administrator can specify whether certain override configuration files should be copied despite being on the `config_files` list.  A pleasant way to still keep Jinja flexibility is for the copied override configuration files to be processed as templates too.  For instance I could copy the `multimap.conf` template to a file in the overrides directory, change it and let the start script process the new template.  This is nicely used in Kolla Ansible for instance.  If it's something relatively straightforward like this I'd be happy to make a PR for it, but might take some time since it's my first issue/PR.\n", "before_files": [{"content": "#!/usr/bin/env python3\n\nimport os\nimport glob\nimport logging as log\nimport requests\nimport shutil\nimport sys\nimport time\nfrom socrate import system,conf\n\nenv = system.set_env()\n\n# Actual startup script\n\nconfig_files = []\nfor rspamd_file in glob.glob(\"/conf/*\"):\n    conf.jinja(rspamd_file, env, os.path.join(\"/etc/rspamd/local.d\", os.path.basename(rspamd_file)))\n    config_files.append(os.path.basename(rspamd_file))\n\nfor override_file in glob.glob(\"/overrides/*\"):\n    if os.path.basename(override_file) not in config_files:\n        shutil.copyfile(override_file, os.path.join(\"/etc/rspamd/local.d\", os.path.basename(override_file)))\n\n# Admin may not be up just yet\nhealthcheck = f'http://{env[\"ADMIN_ADDRESS\"]}:8080/internal/rspamd/local_domains'\nwhile True:\n    time.sleep(1)\n    try:\n        if requests.get(healthcheck,timeout=2).ok:\n            break\n    except:\n        pass\n    log.warning(\"Admin is not up just yet, retrying in 1 second\")\n\n# Run rspamd\nos.system(\"mkdir -m 755 -p /run/rspamd\")\nos.system(\"chown rspamd:rspamd /run/rspamd\")\nos.system(\"find /var/lib/rspamd | grep -v /filter | xargs -n1 chown rspamd:rspamd\")\nos.execv(\"/usr/bin/rspamd\", [\"rspamd\", \"-f\", \"-u\", \"rspamd\", \"-g\", \"rspamd\"])\n", "path": "core/rspamd/start.py"}], "after_files": [{"content": "#!/usr/bin/env python3\n\nimport os\nimport glob\nimport logging as log\nimport requests\nimport shutil\nimport sys\nimport time\nfrom socrate import system,conf\n\nenv = system.set_env()\n\n# Actual startup script\n\nconfig_files = []\nfor rspamd_file in glob.glob(\"/conf/*\"):\n    conf.jinja(rspamd_file, env, os.path.join(\"/etc/rspamd/local.d\", os.path.basename(rspamd_file)))\n    if rspamd_file != '/conf/forbidden_file_extension.map':\n        config_files.append(os.path.basename(rspamd_file))\n\nfor override_file in glob.glob(\"/overrides/*\"):\n    if os.path.basename(override_file) not in config_files:\n        shutil.copyfile(override_file, os.path.join(\"/etc/rspamd/local.d\", os.path.basename(override_file)))\n\n# Admin may not be up just yet\nhealthcheck = f'http://{env[\"ADMIN_ADDRESS\"]}:8080/internal/rspamd/local_domains'\nwhile True:\n    time.sleep(1)\n    try:\n        if requests.get(healthcheck,timeout=2).ok:\n            break\n    except:\n        pass\n    log.warning(\"Admin is not up just yet, retrying in 1 second\")\n\n# Run rspamd\nos.system(\"mkdir -m 755 -p /run/rspamd\")\nos.system(\"chown rspamd:rspamd /run/rspamd\")\nos.system(\"find /var/lib/rspamd | grep -v /filter | xargs -n1 chown rspamd:rspamd\")\nos.execv(\"/usr/bin/rspamd\", [\"rspamd\", \"-f\", \"-u\", \"rspamd\", \"-g\", \"rspamd\"])\n", "path": "core/rspamd/start.py"}]}