title: How to bypass Incapsula with Python
body: <p>I use <a href="https://scrapy.org/" rel="noreferrer">Scrapy</a> and I try to scrape <a href="https://courses-en-ligne.carrefour.fr/" rel="noreferrer">this site</a> that uses <strong>Incapsula</strong> </p>
  
  <pre><code>&lt;meta name="robots" content="noindex,nofollow"&gt;
  &lt;script src="/_Incapsula_Resource?SWJIYLWA=719d34d31c8e3a6e6fffd425f7e032f3"&gt;
  &lt;/script&gt;
  </code></pre>
  
  <p>I had already asked a <a href="https://stackoverflow.com/questions/39354465/why-scrapy-returns-an-iframe"><strong>Question</strong></a> about this issue  2 years ago, but this method (<a href="https://github.com/ziplokk1/incapsula-cracker-py3" rel="noreferrer">Incapsula-Cracker</a>) does not work anymore.</p>
  
  <p>I tried to understand <strong><a href="https://pythonexample.com/snippet/python/incapsulapy_yakudzam_python" rel="noreferrer">How Incapsula works</a></strong>  and I tried this for bypass it </p>
  
  <pre><code>def start_requests(self):
      yield Request('https://courses-en-ligne.carrefour.fr',  cookies={'store': 92}, dont_filter=True, callback = self.init_shop)
  def init_shop(self,response) :
      result_content      = response.body
      RE_ENCODED_FUNCTION = re.compile('var b="(.*?)"', re.DOTALL)
      RE_INCAPSULA        = re.compile('(_Incapsula_Resource\?SWHANEDL=.*?)"')
      INCAPSULA_URL       = 'https://courses-en-ligne.carrefour.fr/%s'
      encoded_func        = RE_ENCODED_FUNCTION.search(result_content).group(1)
      decoded_func        = ''.join([chr(int(encoded_func[i:i+2], 16)) for i in xrange(0, len(encoded_func), 2)])
      incapsula_params    = RE_INCAPSULA.search(decoded_func).group(1)
      incap_url           = INCAPSULA_URL % incapsula_params
      yield Request(incap_url)
  def parse(self):
      print response.body 
  </code></pre>
  
  <p>But i'm redirected to <a href="https://courses-en-ligne.carrefour.fr/_Incapsula_Resource?CWUDNSAI=27&amp;xinfo=3-10784678-0%200NNN%20RT%281523525225370%20394%29%20q%280%20-1%20-1%20-1%29%20r%280%20-1%29%20B12%284%2C316%2C0%29%20U10000&amp;incident_id=459000960022408474-41333502566401539&amp;edet=12&amp;cinfo=04000000" rel="noreferrer">RE-Captcha Page</a> </p>
  
  <pre><code>&lt;html style="height:100%"&gt;
  &lt;head&gt;
  &lt;META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"&gt;
  &lt;meta name="format-detection" content="telephone=no"&gt;
  &lt;meta name="viewport" content="initial-scale=1.0"&gt;
  &lt;meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"&gt;
  &lt;/head&gt;
  &lt;body style="margin:0px;height:100%"&gt;
  &lt;iframe src="/_Incapsula_Resource?CWUDNSAI=27&amp;xinfo=3-10784678-0%200NNN%20RT%281523525225370%20394%29%20q%280%20-1%20-1%20-1%29%20r%280%20-1%29%20B12%284%2c316%2c0%29%20U10000&amp;incident_id=459000960022408474-41333502566401539&amp;edet=12&amp;cinfo=04000000" frameborder=0 width="100%" height="100%" marginheight="0px" marginwidth="0px"&gt;Request unsuccessful. Incapsula incident ID: 459000960022408474-41333502566401539
  &lt;/iframe&gt;
  &lt;/body&gt;
  &lt;/html&gt;
  </code></pre>