title: My Scrapy/Elasticsearch script returns "Unable to serialize" error
body: <p>So guys, for the past 18 hours, I've desperately been trying to find a workaround for a bug in my code, and I think it's time for me to seek for some help.</p>
  
  <p>I'm building a web scraper, its goal is to download a page, grab anchor texts, internal links, referrer URL, and save data to DB. Here's the relevant part of my Scrapy code;</p>
  
  <pre><code>def parse_items(self, response):
      item = InternallinkItem()
  
      # Current URL
      item["current_url"] = response.url
  
      # get anchor text and clean it
      anchor = response.meta.get('link_text')
      item["anchor_text"] = " ".join(anchor.split())
  
  
     # get the referrer URL (Problem is here)
     referring_url = response.request.headers.get('Referer')
     item["referring_url"] = referring_url
  
  
      yield item
  </code></pre>
  
  <p>Technologies I use are Python, Scrapy, Elasticsearch. They are all up to date and my dev environment is Windows. When I run the code above, I'm faced with this error; </p>
  
  <pre><code>raise TypeError("Unable to serialize %r (type: %s)" % (data, type(data)))
  
  TypeError: Unable to serialize b'https://example.com' (type: &lt;class 'bytes'&gt;)
  </code></pre>
  
  <p>So, after so many trial and error, I was able to track it down and pinpoint the issue. When I remove the part that grabs the referrer URL, everything works just fine. It gets the data I want and saves to Elasticsearch successfully. </p>
  
  <p>As someone who's fairly new to programming, I have no idea how to proceed. </p>
  
  <p>I tried, grabbing referrer URL in some other way, didn't work.. </p>
  
  <p>Tried writing my own pipeline, instead of using scrapy-elasticsearch library, but got the same error, also gave a shot to changing type from byte to STR, well you guessed it right, it didn't work either. </p>
  
  <p>Any help would be highly appreciated as I am really stuck here!</p>
  
  <p>EDIT:
  My settings.py file;</p>
  
  <pre><code>ITEM_PIPELINES = {
      'scrapyelasticsearch.scrapyelasticsearch.ElasticSearchPipeline': 500
  }
  
  ELASTICSEARCH_SERVERS = ['localhost'] 
  ELASTICSEARCH_INDEX = 'myindex'
  ELASTICSEARCH_TYPE = 'internallink'
  #ELASTICSEARCH_UNIQ_KEY = ['current_url']
  </code></pre>