title: the order of Scrapy Crawling URLs with long start_urls list and urls yiels from spider
body: <p>Help! Reading the source code of <code>Scrapy</code> is not easy for me.
  I have a very long <code>start_urls</code> list. it is about 3,000,000 in a file. So,I make the <code>start_urls</code> like this:</p>
  
  <pre><code>start_urls = read_urls_from_file(u"XXXX")
  def read_urls_from_file(file_path):
      with codecs.open(file_path, u"r", encoding=u"GB18030") as f:
          for line in f:
              try:
                  url = line.strip()
                  yield url
              except:
                  print u"read line:%s from file failed!" % line
                  continue
      print u"file read finish!"
  </code></pre>
  
  <p>MeanWhile, my spider's callback functions are like this：</p>
  
  <pre><code>  def parse(self, response):
          self.log("Visited %s" % response.url)
          return  Request(url=("http://www.baidu.com"), callback=self.just_test1)
      def just_test1(self, response):
          self.log("Visited %s" % response.url)
          return Request(url=("http://www.163.com"), callback=self.just_test2)
      def just_test2(self, response):
          self.log("Visited %s" % response.url)
          return []
  </code></pre>
  
  <p>my questions are:</p>
  
  <ol>
  <li>the order of the urls used by downloader? Will the requests made by
  <code>just_test1</code>,<code>just_test2</code>  be used by downloader only after the all
  <code>start_urls</code> are used?(I have made some tests, it seems that the
  answer is No)</li>
  <li>What decides the order? Why and How is this order? How can we control it?</li>
  <li>Is this a good way to deal with so many urls which are already in a file? What else?  </li>
  </ol>
  
  <p>Thank you very much!!!</p>
  
  <p>Thanks for answers.But I am still a bit confused:
  <a href="http://doc.scrapy.org/en/latest/faq.html#does-scrapy-crawl-in-breath-first-or-depth-first-order" rel="noreferrer">By default, Scrapy uses a LIFO queue for storing pending requests.</a></p>
  
  <ol>
  <li>The <code>requests</code> made by spiders' callback function will be given to the
  <code>scheduler</code>.Who does the same thing to <code>start_url's requests</code>?The spider
  <code>start_requests()</code> function only generate an iterator without giving
  the real requests. </li>
  <li>Will the all <code>requests</code>(start_url's and callback's) be in the same request's queue?How many queues are there in <code>Scrapy</code>?</li>
  </ol>