<h3 dir="auto">Description</h3>
<p dir="auto">I send some <code class="notranslate">custom_settings</code> values through api. While these values are successfully received in kwargs, they are not applied in crawler settings.<br>
I wrote two different codes for the same thing, but I didn't succeed, and maybe scrapy has a bug in this part or I made a mistake.</p>
<h3 dir="auto">First code:</h3>
<pre class="notranslate">from scrapy.spiders import CrawlSpider, Rule
from scrapy.loader import ItemLoader
from kavoush.lxmlhtml import LxmlLinkExtractor as LinkExtractor
from kavoush.items import PageLevelItem

class PageSpider(CrawlSpider):
  name = 'github'

  def __init__(self, *args, **kwargs):
    self.start_urls = kwargs.get('host_name')
    self.allowed_domains = [self.start_urls]
    self.custom_settings = {
            'CONCURRENT_REQUESTS': int(kwargs.get('CONCURRENT_REQUESTS')),
            'ROBOTSTXT_OBEY': kwargs.get('ROBOTSTXT_OBEY') == 'True',
        }
    self.logger.info(f'CONCURRENT_REQUESTS? {self.custom_settings}')

    self.rules = (
            Rule(LinkExtractor(allow=(self.start_urls),deny=('\.webp'),unique=True),
            callback='parse',
            follow=True),
      )
    super(PageSpider, self).__init__(*args, **kwargs)

  def parse(self,response):
        loader = ItemLoader(item=PageLevelItem(), response=response)
        loader.add_xpath('page_source_html_lang', "//html/@lang")
        yield loader.load_item()

  def errback_domain(self, failure):
    self.logger.error(repr(failure))
</pre>
<br>
And according to the photo, the settings I want via api are not available in Overridden settings, but in the log, the correct values are sent and received successfully.
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/f760269ab9a913b5d2ec5e2d8da1200f12c0485420f24f268cf6b38cf25ab9ff/68747470733a2f2f692e6962622e636f2f503670677478322f7070312e706e67"><img src="https://camo.githubusercontent.com/f760269ab9a913b5d2ec5e2d8da1200f12c0485420f24f268cf6b38cf25ab9ff/68747470733a2f2f692e6962622e636f2f503670677478322f7070312e706e67" data-canonical-src="https://i.ibb.co/P6pgtx2/pp1.png" style="max-width: 100%;"></a></p>
<br>
The photo below is the request that I sent through api and with the help of scrapyd.
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/5a3017e8cc3be03519b5946e322eda7ff247c3f95b21d377602fa73113e2bc75/68747470733a2f2f692e6962622e636f2f4c3643536767442f617069696e2e706e67"><img src="https://camo.githubusercontent.com/5a3017e8cc3be03519b5946e322eda7ff247c3f95b21d377602fa73113e2bc75/68747470733a2f2f692e6962622e636f2f4c3643536767442f617069696e2e706e67" data-canonical-src="https://i.ibb.co/L6CSggD/apiin.png" style="max-width: 100%;"></a></p>
<br>
The second code has the same problem, actually I expected one of the codes (the first code or the second code) to work, but none of them applies the settings.<br>
<h3 dir="auto">Second code:</h3>
<pre class="notranslate">from scrapy.spiders import CrawlSpider, Rule
from scrapy.loader import ItemLoader
from kavoush.lxmlhtml import LxmlLinkExtractor as LinkExtractor
from kavoush.items import PageLevelItem
from scrapy.settings import Settings

class PageSpider(CrawlSpider):
  name = 'github1'

  def __init__(self, *args, **kwargs):
    self.start_urls = kwargs.get('host_name')
    self.allowed_domains = [self.start_urls]
    self.custom_settings = {
            'CONCURRENT_REQUESTS': int(kwargs.get('CONCURRENT_REQUESTS')),
            'ROBOTSTXT_OBEY': kwargs.get('ROBOTSTXT_OBEY') == 'True',
        }
    self.logger.info(f'CONCURRENT_REQUESTS? {self.custom_settings}')

    for key, value in kwargs.items():
        if key in self.custom_settings:
            self.custom_settings[key] = value

    settings = Settings()
    settings.setdict(self.custom_settings, priority='spider')
    self.settings = settings

    self.rules = (
            Rule(LinkExtractor(allow=(self.start_urls),deny=('\.webp'),unique=True),
            callback='parse',
            follow=True),
      )
    super(PageSpider, self).__init__(*args, **kwargs)

  def parse(self,response):
        loader = ItemLoader(item=PageLevelItem(), response=response)
        loader.add_xpath('page_source_html_lang', "//html/@lang")
        yield loader.load_item()

  def errback_domain(self, failure):
    self.logger.error(repr(failure))
</pre>
<br>
<h3 dir="auto">Versions</h3>
<pre class="notranslate">Scrapy       : 2.7.1
lxml         : 4.9.2.0
libxml2      : 2.9.14
cssselect    : 1.2.0
parsel       : 1.7.0
w3lib        : 2.1.1
Twisted      : 22.10.0
Python       : 3.10.9 
pyOpenSSL    : 22.1.0 (OpenSSL 3.0.7 1 Nov 2022)
cryptography : 38.0.1
Platform     : Linux-5.15.0-50-generic-x86_64-with-glibc2.35
</pre>
<h3 dir="auto">Additional context</h3>
<p dir="auto">I already asked my question in stackoverflow and said maybe I don't know, but the answer in stackoverflow was similar to my first code, but it doesn't work in scrapy.<br>
In short, I want to send some <code class="notranslate">custom_settings</code> via api, but I didn't succeed.<br>
Thank you friends.</p>
<p dir="auto">This issue might be related to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="88418139" data-permission-text="Title is private" data-url="https://github.com/scrapy/scrapy/issues/1305" data-hovercard-type="issue" data-hovercard-url="/scrapy/scrapy/issues/1305/hovercard" href="https://github.com/scrapy/scrapy/issues/1305">#1305</a></p>
<p dir="auto">I noticed that <code class="notranslate">settings</code> are frozen in <a href="https://github.com/scrapy/scrapy/blob/master/scrapy/crawler.py#L57">https://github.com/scrapy/scrapy/blob/master/scrapy/crawler.py#L57</a><br>
However, in a given project I had a requirement to change some settings based on some spider arguments. An alternative would be to write this spider as a base class and extend it from specific spiders setting the proper <code class="notranslate">settings</code>.<br>
However, I think it would make sense to only freeze settings after the spider and other components were initialized. Or, provide some other entry point to configure settings based on arguments.<br>
The other option is to use <code class="notranslate">-s</code> arguments, but in my case I was changing the <code class="notranslate">FEED_EXPORT_FIELDS</code> setting (<a href="https://docs.scrapy.org/en/latest/topics/feed-exports.html#std:setting-FEED_EXPORT_FIELDS" rel="nofollow">https://docs.scrapy.org/en/latest/topics/feed-exports.html#std:setting-FEED_EXPORT_FIELDS</a>).</p>
<p dir="auto">Any thoughts here?</p>