<p dir="auto">My spider is scraping on a big website, so I start it with the pausing and resuming crawls setup.</p>
<p dir="auto"><code class="notranslate">scrapy crawl somespider -s JOBDIR=crawls/somespider-1</code></p>
<p dir="auto">But every time I pause and resume later, I find that after the resume the spider will revisit all the website in the start_urls again. Is it a bug ?</p>
<h3 dir="auto">Description</h3>
<p dir="auto">I tried following the docs here <a href="https://docs.scrapy.org/en/latest/topics/feed-exports.html#scrapy.extensions.feedexport.uri_params" rel="nofollow">https://docs.scrapy.org/en/latest/topics/feed-exports.html#scrapy.extensions.feedexport.uri_params</a> to have a dynamic Feed name.<br>
Here is the relevant code:</p>
<p dir="auto"><code class="notranslate">uri_params</code> function:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from urllib.parse import urlsplit
from pathlib import PurePath


def source_params(params, spider):
    return {**params, &quot;source_to_file&quot;: PurePath(urlsplit(spider.source).path).name}"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">urllib</span>.<span class="pl-s1">parse</span> <span class="pl-k">import</span> <span class="pl-s1">urlsplit</span>
<span class="pl-k">from</span> <span class="pl-s1">pathlib</span> <span class="pl-k">import</span> <span class="pl-v">PurePath</span>


<span class="pl-k">def</span> <span class="pl-en">source_params</span>(<span class="pl-s1">params</span>, <span class="pl-s1">spider</span>):
    <span class="pl-k">return</span> {<span class="pl-c1">**</span><span class="pl-s1">params</span>, <span class="pl-s">"source_to_file"</span>: <span class="pl-v">PurePath</span>(<span class="pl-en">urlsplit</span>(<span class="pl-s1">spider</span>.<span class="pl-s1">source</span>).<span class="pl-s1">path</span>).<span class="pl-s1">name</span>}</pre></div>
<p dir="auto">Spider</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import scrapy
from pathlib import Path

&quot;&quot;&quot;Gets all videos from a playlist, should also work for actors&quot;&quot;&quot;


class PlaylistsSpider(scrapy.Spider):
    name = &quot;playlists&quot;
    custom_settings = {
        &quot;FEEDS&quot;: {
            Path(&quot;%(source_to_file)s&quot;).with_suffix(&quot;.csv&quot;).name: {&quot;format&quot;: &quot;csv&quot;}
        },
        &quot;FEED_URI_PARAMS&quot;: &quot;myproject.utils.source_params&quot;,
    }

    def __init__(self, *args, **kwargs):
        self.source = kwargs.pop(&quot;source&quot;)
        self.start_urls = [self.source]

    def parse(self, response):
        pass"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">scrapy</span>
<span class="pl-k">from</span> <span class="pl-s1">pathlib</span> <span class="pl-k">import</span> <span class="pl-v">Path</span>

<span class="pl-s">"""Gets all videos from a playlist, should also work for actors"""</span>


<span class="pl-k">class</span> <span class="pl-v">PlaylistsSpider</span>(<span class="pl-s1">scrapy</span>.<span class="pl-v">Spider</span>):
    <span class="pl-s1">name</span> <span class="pl-c1">=</span> <span class="pl-s">"playlists"</span>
    <span class="pl-s1">custom_settings</span> <span class="pl-c1">=</span> {
        <span class="pl-s">"FEEDS"</span>: {
            <span class="pl-v">Path</span>(<span class="pl-s">"%(source_to_file)s"</span>).<span class="pl-en">with_suffix</span>(<span class="pl-s">".csv"</span>).<span class="pl-s1">name</span>: {<span class="pl-s">"format"</span>: <span class="pl-s">"csv"</span>}
        },
        <span class="pl-s">"FEED_URI_PARAMS"</span>: <span class="pl-s">"myproject.utils.source_params"</span>,
    }

    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
        <span class="pl-s1">self</span>.<span class="pl-s1">source</span> <span class="pl-c1">=</span> <span class="pl-s1">kwargs</span>.<span class="pl-en">pop</span>(<span class="pl-s">"source"</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">start_urls</span> <span class="pl-c1">=</span> [<span class="pl-s1">self</span>.<span class="pl-s1">source</span>]

    <span class="pl-k">def</span> <span class="pl-en">parse</span>(<span class="pl-s1">self</span>, <span class="pl-s1">response</span>):
        <span class="pl-k">pass</span></pre></div>
<h3 dir="auto">Steps to Reproduce</h3>
<p dir="auto">During debugging and breaking at this line </p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/scrapy/scrapy/blob/cbf934955223c29f0632fbba94d9586849a3b744/scrapy/extensions/feedexport.py#L542">scrapy/scrapy/extensions/feedexport.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 542
      in
      <a data-pjax="true" class="commit-tease-sha" href="/scrapy/scrapy/commit/cbf934955223c29f0632fbba94d9586849a3b744">cbf9349</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L542" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="542"></td>
          <td id="LC542" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">return</span> <span class="pl-s1">params</span> </td>
        </tr>
    </tbody></table>
  </div>
</div>
<br>
this happens:<p></p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="-&gt; params['source_to_files']
Traceback (most recent call last):
  File &quot;&lt;string&gt;&quot;, line 1, in &lt;module&gt;
KeyError: 'source_to_files'"><pre class="notranslate"><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-s1">params</span>[<span class="pl-s">'source_to_files'</span>]
<span class="pl-v">Traceback</span> (<span class="pl-s1">most</span> <span class="pl-s1">recent</span> <span class="pl-s1">call</span> <span class="pl-s1">last</span>):
  <span class="pl-v">File</span> <span class="pl-s">"&lt;string&gt;"</span>, <span class="pl-s1">line</span> <span class="pl-c1">1</span>, <span class="pl-s1">in</span> <span class="pl-c1">&lt;</span><span class="pl-s1">module</span><span class="pl-c1">&gt;</span>
<span class="pl-v">KeyError</span>: <span class="pl-s">'source_to_files'</span></pre></div>
<p dir="auto">and indeed there's no file output if I let it run</p>
<p dir="auto"><strong>Expected behavior:</strong><br>
Export according to file name as specified by <code class="notranslate">FEEDS</code> and <code class="notranslate">URI_PARAMS</code><br>
<strong>Actual behavior:</strong><br>
No feed export<br>
<strong>Reproduces how often:</strong><br>
Until I replaced this line: </p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/scrapy/scrapy/blob/cbf934955223c29f0632fbba94d9586849a3b744/scrapy/extensions/feedexport.py#L541">scrapy/scrapy/extensions/feedexport.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 541
      in
      <a data-pjax="true" class="commit-tease-sha" href="/scrapy/scrapy/commit/cbf934955223c29f0632fbba94d9586849a3b744">cbf9349</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L541" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="541"></td>
          <td id="LC541" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-en">uripar_function</span>(<span class="pl-s1">params</span>, <span class="pl-s1">spider</span>) </td>
        </tr>
    </tbody></table>
  </div>
</div>
<br>
with the following:<p></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="        params = uripar_function(params, spider)"><pre class="notranslate"><code class="notranslate">        params = uripar_function(params, spider)
</code></pre></div>
<h3 dir="auto">Versions</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Scrapy       : 2.5.0
lxml         : 4.6.3.0
libxml2      : 2.9.10
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 21.7.0
Python       : 3.9.6 (default, Jun 29 2021, 05:25:02) - [Clang 12.0.5 (clang-1205.0.22.9)]
pyOpenSSL    : 21.0.0 (OpenSSL 1.1.1l  24 Aug 2021)
cryptography : 35.0.0
Platform     : macOS-11.4-x86_64-i386-64bit"><pre class="notranslate"><code class="notranslate">Scrapy       : 2.5.0
lxml         : 4.6.3.0
libxml2      : 2.9.10
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 21.7.0
Python       : 3.9.6 (default, Jun 29 2021, 05:25:02) - [Clang 12.0.5 (clang-1205.0.22.9)]
pyOpenSSL    : 21.0.0 (OpenSSL 1.1.1l  24 Aug 2021)
cryptography : 35.0.0
Platform     : macOS-11.4-x86_64-i386-64bit
</code></pre></div>