title: scrapy python find href references recursively
body: <p>I'm trying to do an scrapy that find and print all href  from  a  start page:</p>
  
  <pre><code>class Ejercicio2(scrapy.Spider):
      name = "Ejercicio2"
      Ejercicio2 = {}
      category = None
      lista_urls =[] #defino una lista para meter las urls
  
  def __init__(self, *args, **kwargs):
      super(Ejercicio2, self).__init__(*args, **kwargs)
      self.start_urls = ['http://www.masterdatascience.es/']
      self.allowed_domains = ['www.masterdatascience.es/']
      url = ['http://www.masterdatascience.es/']
  
  
  def parse(self, response):
      print(response)
      # hay_enlace=response.css('a::attr(href)')
      # if hay_enlace:
      links = response.xpath("a/@href")
      for el in links:
          url = response.css('a::attr(href)').extract()
          print(url)
          next_url = response.urljoin(el.xpath("a/@href").extract_first())
          print(next_url)
          print('pasa por aqui')
          yield scrapy.Request(url, self.parse())
          # yield scrapy.Request(next_url, callback=self.parse)
          print(next_url)
  </code></pre>
  
  <p>But is not working as expected, not following the "href" references encountered,only the first one.</p>