title: How to get AppEngine map reduce to scale out?
body: <p>I have written a simple MapReduce flow to read in lines from a CSV from a file on Google Cloud Storage and subsequently make an Entity.  However, <strong>I can't seem to get it to run on more than one shard</strong>.</p>
  
  <p>The code makes use of mapreduce.control.start_map and looks something like this.</p>
  
  <pre><code>class LoadEntitiesPipeline(webapp2.RequestHandler):
          id = control.start_map(map_name,
                            handler_spec="backend.line_processor",
                            reader_spec="mapreduce.input_readers.FileInputReader",
                            queue_name=get_queue_name("q-1"),
                            shard_count=shard_count,
                            mapper_parameters={
                                'shard_count': shard_count,
                                'batch_size': 50,
                                'processing_rate': 1000000,
                                'files': [gsfile],
                                'format': 'lines'})
  </code></pre>
  
  <p>I have shard_count in both places, because I'm not sure what methods actually need it.  Setting shard_count anywhere from 8 to 32, doesn't change anything as the status page always says 1/1 shards running. To separate things, I've made everything run on a backend queue with a large number of instances.  I've tried adjusting the queue parameters <a href="https://code.google.com/p/appengine-mapreduce/wiki/InstancesQueuesShardsAndSlices">per this wiki</a>.  In the end, it seems to just run serially.</p>
  
  <p>Any ideas? Thanks!</p>
  
  <p><strong>Update (Still no success):</strong></p>
  
  <p>In trying to isolate things, I tried making the call using direct calls to pipeline like so:</p>
  
  <pre><code>class ImportHandler(webapp2.RequestHandler):
  
      def get(self, gsfile):
          pipeline = LoadEntitiesPipeline2(gsfile)
          pipeline.start(queue_name=get_queue_name("q-1"))
  
          self.redirect(pipeline.base_path + "/status?root=" + pipeline.pipeline_id)
  
  
  class LoadEntitiesPipeline2(base_handler.PipelineBase):
  
      def run(self, gsfile):
          yield mapreduce_pipeline.MapperPipeline(
             'loadentities2_' + gsfile,
             'backend.line_processor',
             'mapreduce.input_readers.FileInputReader',
             params={'files': [gsfile], 'format': 'lines'},
             shards=32
          )
  </code></pre>
  
  <p>With this new code, it still only runs on one shard.  <strong>I'm starting to wonder if mapreduce.input_readers.FileInputReader is capable of parallelizing input by line.</strong></p>