title: Need help on figuring out how to scrape real time streaming data with Python
body: <p>I was trying to scrape the Current Usage for this local web server. This number updates every second with values that are generated from a random number generator.</p>
  <blockquote>
  <p>Current Time: 07:25:16 UTC</p>
  <p>Current Date: 2018-11-28 UTC</p>
  <p>Current Usage: 13 kW</p>
  </blockquote>
  <p>This is what I've tried with Beautifulsoup so far:</p>
  <pre><code>import requests
  from bs4 import BeatifulSoup
  import time
  
  def get_count():
    url = &quot;http://10.0.0.206/apps/cy8ckit_062_demo/main.html&quot;
  
    # request with fake header, otherwise you will get an 403 HTTP error
    r = request.get(url, headers={'User-Agent': Mozilla/5.0})
  
  while True:
    print(get_count())
    time.sleep(8)
  </code></pre>
  <p>However, when I run this script, I get an output 'None' every 8 seconds</p>
  <p>Here is the output of the inspection of the web server:</p>
  <blockquote>
  <p>Current Time: 07:39:42 UTC</p>
  <p>Current Date 2018-11-28 UTC</p>
  <p>Current Usage: 8 kW</p>
  </blockquote>
  <p>I've been trying to follow this: <a href="https://stackoverflow.com/questions/39489168/how-to-scrape-real-time-streaming-data-with-python/39489328">How to scrape real time streaming data with Python?</a></p>
  <p>This is what I get as an output after trying @chitown88 code:</p>
  <pre><code>Traceback (most recent call last):
    File &quot;C:/seniord/csusite/readweb.py&quot;, line 14, in &lt;module&gt;
      soup = BeautifulSoup(r.text, 'html.parser')
  NameError: name 'r' is not defined
  </code></pre>
  <p>After trying the revised code from @chitown88, I get this as an output(Doesn't show the dynamic values but I thought beautifulsoup fixes that issue):</p>
  <pre><code>&lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot;?&gt;
  &lt;!DOCTYPE html PUBLIC &quot;-//W3C//DTD XHTML 1.0 Strict//EN&quot;
  &quot;http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd&quot;&gt;
  
  &lt;html lang=&quot;en&quot; xml:lang=&quot;en&quot; xmlns=&quot;http://www.w3.org/1999/xhtml&quot;&gt;
  &lt;head&gt;
  &lt;link href=&quot;../../styles/buttons.css&quot; rel=&quot;stylesheet&quot; type=&quot;text/css&quot;/&gt;
  &lt;title&gt;CE222494 PSoC 6 WICED WiFi Demo&lt;/title&gt;
  &lt;meta content=&quot;text/html; charset=utf-8&quot; http-equiv=&quot;Content-Type&quot;/&gt;
  &lt;script src=&quot;../../scripts/general_ajax_script.js&quot; type=&quot;text/javascript&quot;&gt;&lt;/script&gt;
  &lt;script type=&quot;text/javascript&quot;&gt;
      /* &lt;![CDATA[ */
         function reloadData()
         {
           do_ajax('/temp_report.html', ajax_handler);
           timeoutID = setTimeout('reloadData()', 500);
         }
        function ajax_handler( result, data )
        {
          switch( result )
          {
              case AJAX_PARTIAL_PROGRESS:
                  break;
              case AJAX_STARTING:
                  break;
              case AJAX_FINISHED:
                  document.getElementById(&quot;currentData&quot;).innerHTML = data;
                  break;
              case AJAX_NO_BROWSER_SUPPORT:
                  document.getElementById(&quot;currentData&quot;).innerHTML = &quot;Failed - your browser does not support this script&quot;;
                  break;
              case AJAX_FAILED:
                  document.getElementById(&quot;currentData&quot;).innerHTML = &quot;There was a problem retrieving data&quot;;
                  break;
          }
        }
  
      /* ]]&gt; */
      &lt;/script&gt;
  &lt;/head&gt;
  &lt;body onload=&quot;reloadData()&quot;&gt;
  &lt;div id=&quot;currentData&quot;&gt;Retrieving current usage data...
      &lt;/div&gt;
  &lt;/body&gt;
  &lt;/html&gt;
  </code></pre>