title: How to use Scrapy's MapCompose input processor
body: <p>In the Scrapy shell for the website <a href="http://www.apkmirror.com/apk/google-inc/sheets/sheets-1-7-152-06-release/google-sheets-1-7-152-06-30-android-apk-download/" rel="nofollow noreferrer">http://www.apkmirror.com/apk/google-inc/sheets/sheets-1-7-152-06-release/google-sheets-1-7-152-06-30-android-apk-download/</a>, I'm trying to parse the <code>version_name</code> (<code>1.7.152.06.30</code>) and <code>version_code</code> (<code>7152063</code>) (see screenshot below) in a succinct way using Scrapy's <a href="https://doc.scrapy.org/en/latest/topics/loaders.html#scrapy.loader.processors.MapCompose" rel="nofollow noreferrer">MapCompose</a> processor.</p>
  
  <p><a href="https://i.stack.imgur.com/DnizZ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/DnizZ.png" alt="enter image description here"></a></p>
  
  <p>My first step is to get all the text from the 'APK details' section:</p>
  
  <pre><code>In [2]: apk_details = response.xpath('//*[@title="APK details"]/following-sibling::*[@class="appspec-value"]//text()').extract()
  </code></pre>
  
  <p>upon which <code>apk_details</code> is a list as follows:</p>
  
  <pre><code>[u'Version: 1.7.152.06.30 (71520630)',
   u'arm ',
   u'Package: com.google.android.apps.docs.editors.sheets',
   u'\n',
   u'191 downloads ']
  </code></pre>
  
  <p>I've defined the following auxiliary functions:</p>
  
  <pre><code>import re
  
  def get_version_line(apk_details):
      '''Get the line containing the version from the 'APK details' section.'''
      return next(line for line in apk_details if line.startswith("Version:"))
  
  def parse_version_line(version_line):
      '''Parse the 'versionName' and 'versionCode' from the relevant line in 'APK details'.'''
      PATTERN = r"^Version: (?P&lt;version_name&gt;.+) \((?P&lt;version_code&gt;\d+)\)\s*$"       # Note that the pattern includes the end-of-line character ($). This is necessary because some package names (e.g. Google Play) themselves contain brackets.
      return re.match(PATTERN, version_line).groupdict()
  </code></pre>
  
  <p>such that the <code>version_name</code> can be obtained as follows:</p>
  
  <pre><code>In [4]: version_line = get_version_line(apk_details)
  
  In [5]: version_line
  Out[5]: u'Version: 1.7.152.06.30 (71520630)'
  
  In [6]: groups = parse_version_line(version_line)
  
  In [7]: groups
  Out[7]: {'version_code': u'71520630', 'version_name': u'1.7.152.06.30'}
  
  In [8]: version_name = groups.get("version_name")
  
  In [9]: version_name
  Out[9]: u'1.7.152.06.30'
  </code></pre>
  
  <p>In other words, I'd like to apply <code>get_version_line</code>, <code>parse_version_line</code>, and <code>lambda d: d.get("version_name")</code> successively to <code>apk_details</code>. However, if I try the following:</p>
  
  <pre><code>In [10]: proc = MapCompose(get_version_line, parse_version_line)
  
  In [11]: proc(apk_details)
  </code></pre>
  
  <p>I get a <code>StopIteration</code> exception:</p>
  
  <pre><code>---------------------------------------------------------------------------
  StopIteration                             Traceback (most recent call last)
  &lt;ipython-input-11-59a0bd60721d&gt; in &lt;module&gt;()
  ----&gt; 1 proc(apk_details)
  
  /usr/local/lib/python2.7/dist-packages/scrapy/loader/processors.pyc in __call__(self, value, loader_context)
       26             next_values = []
       27             for v in values:
  ---&gt; 28                 next_values += arg_to_iter(func(v))
       29             values = next_values
       30         return values
  
  /home/kurt/dev/apkmirror_scraper/apkmirror_scraper/items.pyc in get_version_line(apk_details)
       33 def get_version_line(apk_details):
       34     '''Get the line containing the version from the 'APK details' section.'''
  ---&gt; 35     return next(line for line in apk_details if line.startswith("Version:"))
       36 
       37 def get_architectures_line(apk_details):
  
  StopIteration:
  </code></pre>
  
  <p>How would I correctly use <code>MapCompose</code> in this case?</p>