title: Getting javascript variable value while scraping with python
body: <p>I know this is asked before also, but I am a newbie in scraping and python. Please help me and it would be very much helpful in my learning path.</p>
  
  <p>I am scraping a news site using python with packages such as <strong>Beautiful Soup</strong> and etc.</p>
  
  <p>I am facing difficulty while getting the value of <code>java script</code> variable which is declared in <code>script</code> tag and also it is getting updated there.</p>
  
  <p>Here is the part of HTML page which I am scraping:(containing only script part)</p>
  
  <pre><code>&lt;!-- Eliminate render-blocking JavaScript and CSS in above-the-fold content --&gt;
    &lt;script src="https://ajax.googleapis.com/ajax/libs/jquery/1.8/jquery.min.js"&gt;&lt;/script&gt;
    &lt;script src="https://cdnjs.cloudflare.com/ajax/libs/materialize/0.97.0/js/materialize.min.js"&gt;&lt;/script&gt;
  
    &lt;script type="text/javascript" src="/dist/scripts/index.js"&gt;&lt;/script&gt;
    &lt;script type="text/javascript" src="/dist/scripts/read.js"&gt;&lt;/script&gt;
    &lt;script src="/dist/scripts/jquery.scrolldepth.min.js"&gt;&lt;/script&gt;
    &lt;script type="text/javascript"&gt;
  
      var min_news_id = "d7zlgjdu-1"; // line 1
      function loadMoreNews(){
        $("#load-more-btn").hide();
        $("#load-more-gif").show();
        $.post("/en/ajax/more_news",{'category':'politics','news_offset':min_news_id},function(data){
            data = JSON.parse(data);
            min_news_id = data.min_news_id||min_news_id; // line 2
            $(".card-stack").append(data.html);
        })
        .fail(function(){alert("Error : unable to load more news");})
        .always(function(){$("#load-more-btn").show();$("#load-more-gif").hide();});
      }
      jQuery.scrollDepth();
    &lt;/script&gt;
  </code></pre>
  
  <p>From the above part, I want to get the value of <code>min_news_id</code> in python.
  I should also get the value of same variable if updated from line 2.</p>
  
  <p>Here is how I am doing it:</p>
  
  <pre><code>    self.pattern = re.compile('var min_news_id = (.+?);') // or self.pattern = re.compile('min_news_id = (.+?);')
      page = bs(htmlPage, "html.parser")
      //find all the scripts tag
      scripts = page.find_all("script")
      for script in scripts:
          for line in script:
              scriptString = str(line)
              if "min_news_id" in scriptString:
                  scriptString.replace('"', '\\"')
                  print(scriptString)
                  if(self.pattern.match(str(scriptString))):
                      print("matched")
                      data = self.pattern.match(scriptString)
                      jsVariable = json.loads(data.groups()[0])
                      InShortsScraper.newsOffset = jsVariable
                      print(InShortsScraper.newsOffset)
  </code></pre>
  
  <p>But I am never getting the value of the variable. Is it problem with my regular expression or any other? Please help me.
  Thank You in advance.</p>