title: Scrapy SitemapSpider - How to yield entry from sitemap_filter to parse
body: <p>I'm building a SitemapSpider.
  I'm trying to filter sitemap entries to exclude entries that contain this substring '/p/' in the link:</p>
  <pre><code>&lt;url&gt;
         &lt;loc&gt;https://example.co.za/product-name/p/product-id&lt;/loc&gt;
         &lt;lastmod&gt;2019-08-27&lt;/lastmod&gt;
         &lt;changefreq&gt;daily&lt;/changefreq&gt;
  &lt;/url&gt;
  </code></pre>
  <p>According to the <a href="https://docs.scrapy.org/en/latest/topics/spiders.html" rel="nofollow noreferrer">Scrapy docs</a> , we can define a <code>sitemap_filter</code> function:</p>
  <pre><code>        for entry in entries:
              date_time = datetime.strptime(entry['lastmod'], '%Y-%m-%d')
              if date_time.year &gt;= 2005:
                  yield entry
  </code></pre>
  <p>In my case I'm filtering on <code>entry['loc']</code> instead of <code>entry['lastmod']</code>.</p>
  <p>Unfortunately I haven't found an example that uses <code>sitemap_filter</code> besides the above.</p>
  <pre><code>from scrapy.spiders import SitemapSpider
  
  class mySpider(SitemapSpider)
      name = 'spiderName'
      sitemap_urls = ['https://example.co.za/medias/sitemap']
      # sitemap_rules = [ ('donut/c', 'parse')]
      
      def sitemap_filter(self, entries):
          for entry in entries:
              if '/p/' not in entry['loc']
                  print(entry)
                  yield entry
      def parse(self, response):
          ...
  </code></pre>
  <p>The code runs fine without the <code>sitemap_filter</code> function, but it's not feasible to define all the <code>sitemap_rules</code>.</p>
  <p>When I run the code above it prints the correct sitemap entries, but it seemingly doesn't proceed to the parse function. The log file shows no errors:</p>
  <pre><code>2022-05-10 17:02:00 [scrapy.core.engine] INFO: Spider opened
  2022-05-10 17:02:00 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
  2022-05-10 17:02:00 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
  2022-05-10 17:02:00 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET https://example.co.za/robots.txt&gt; (referer: None)
  2022-05-10 17:02:00 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET https://example.co.za/medias/sitemap.xml&gt; (referer: None)
  2022-05-10 17:02:05 [scrapy.crawler] INFO: Received SIGINT, shutting down gracefully. Send again to force 
  2022-05-10 17:02:06 [scrapy.crawler] INFO: Received SIGINT twice, forcing unclean shutdown
  2022-05-10 17:02:09 [scrapy.core.engine] INFO: Closing spider (shutdown)
  2022-05-10 17:02:09 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
  </code></pre>
  <p>I'm looking for a way to send the entries yielded by <code>sitemap_filter</code> to the <code>parse</code> function, or alternatively, a way to filter sitemap entries before scrapy opens the links.</p>