title: How to convert response.text to json in Scrapy
body: <p>I am scraping the financial data from below link using Scrapy:</p>
  
  <p><a href="http://web.ifzq.gtimg.cn/appstock/hk/HkInfo/getFinReport?type=3&amp;reporttime_type=1&amp;code=00001&amp;startyear=1990&amp;endyear=2016&amp;_callback=jQuery1124033955090772971586_1528569153921&amp;_=1528569153953" rel="nofollow noreferrer">Financial Data from Tencent</a></p>
  
  <p>The reponse.body is like below:</p>
  
  <p><a href="https://i.stack.imgur.com/qwvfX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/qwvfX.png" alt="Response.body"></a></p>
  
  <p>I have tried to split the response using regular regression then convert it to json but it shows no json object, here is my code:</p>
  
  <pre class="lang-py prettyprint-override"><code>import scrapy
  import re
  import json
  
  class StocksSpider(scrapy.Spider):
      name = 'stocks'
      allowed_domains = ['web.ifzq.gtimg.cn']
      start_urls = ['http://web.ifzq.gtimg.cn/appstock/hk/HkInfo/getFinReport?type=3&amp;reporttime_type=1&amp;code=00001&amp;startyear=1990&amp;endyear=2016&amp;_callback=jQuery11240339550$']
  
      def start_requests(self):
          for url in self.start_urls:
              yield scrapy.Request(url=url, callback=self.parse,
              #endpoint='render.json', # optional; default is render.html
              #splash_url='&lt;url&gt;',     # optional; overrides SPLASH_URL
              #slot_policy=scrapy_splash.SlotPolicy.PER_DOMAIN,  # optional
              )
  
      def parse(self, response):
          try:
              json_data = re.search('\{\"data\"\:(.+?)\}\}\]', response.text).group(1)
          except AttributeError:
              json_data = ''
          #print json_data
          loaded_json = json.loads(json_data)
  
          print loaded_json
  
  
  </code></pre>
  
  <p>It throws an error saying that no json object can be decoded:</p>
  
  <pre><code>    Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/defer.py", line 102, in iter_errback
      yield next(it)
    File "/usr/local/lib/python2.7/dist-packages/scrapy_splash/middleware.py", line 156, in process_spider_output
      for el in result:
    File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/offsite.py", line 30, in process_spider_output
      for x in result:
    File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/referer.py", line 339, in &lt;genexpr&gt;
      return (_set_referer(r) for r in result or ())
    File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/urllength.py", line 37, in &lt;genexpr&gt;
      return (r for r in result or () if _filter(r))
    File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/depth.py", line 58, in &lt;genexpr&gt;
      return (r for r in result or () if _filter(r))
    File "/root/finance/finance/spiders/stocks.py", line 25, in parse
      loaded_json = json.loads(json_data)
    File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
      return _default_decoder.decode(s)
    File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
      obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
      raise ValueError("No JSON object could be decoded")
  ValueError: No JSON object could be decoded
  2018-06-09 23:54:26 [scrapy.core.engine] INFO: Closing spider (finished)
  </code></pre>
  
  <p>My goal is to convert it to json so that I can easily iterate the content.
  Is it necessary to convert it to json and how to convert in this case? The response is in unicode format so that I need to convert it to utf-8 as well? Is there any other good way to do iteration?</p>