title: Scraping only new elements from website using requests and bs4
body: <p>I want to look at a certain website to collect data from it, at the first visit to the said website I collect all the data that was there to ignore it. I want to perform a certain action if a new row is added(for example print as in here). But whenever a new item appears it seems to print every single row on the website even though I'm checking if the row exists already in the dictionary. Don't know how to fix it, can anyone take a look?</p>
  <pre class="lang-py prettyprint-override"><code>import requests
  import re
  import copy
  import time
  
  from datetime import date
  from bs4 import BeautifulSoup
  
  
  class KillStatistics:
      def __init__(self):
          self.records = {}
          self.watched_names = [&quot;Test&quot;]
          self.iter = 0
  
      def parse_records(self):
          r = requests.get(&quot;http://149.56.28.71/?subtopic=killstatistics&quot;)
          soup = BeautifulSoup(r.content, &quot;html.parser&quot;)
          table = soup.findChildren(&quot;table&quot;)
  
          for record in table:
              for data in record:
                  if data.text == &quot;Last Deaths&quot;:
                      pass
                  else:
                      entry = data.text
                      entry = re.split(&quot;..?(?=[0-9][A-Z]).&quot;, data.text)
                      entry[0] = entry[0].split(&quot;, &quot;)
                      entry[0][0] = entry[0][0].split(&quot;.&quot;)
  
                      entry_id, day, month, year, hour = (
                          entry[0][0][0],
                          entry[0][0][1],
                          entry[0][0][2],
                          entry[0][0][3],
                          entry[0][1],
                      )
  
                      message = entry[1]
                      nickname = (re.findall(&quot;.+?(?=at)&quot;, message)[0]).strip()
                      killed_by = (re.findall(r&quot;(?&lt;=\bby).*&quot;, message)[0]).strip()
                      if self.iter &lt; 1:
                          &quot;&quot;&quot;Its the first visit to the website, i want it to download all the data and store it in dictionary&quot;&quot;&quot;
                          self.records[
                              entry_id
                          ] = f&quot;{nickname} was killed by {killed_by} at {day}-{month}-{year} {hour}&quot;
                      elif (
                          self.iter &gt; 1
                          and f&quot;{nickname} was killed by {killed_by} at {day}-{month}-{year} {hour}&quot;
                          not in self.records.values()
                      ):
                          &quot;&quot;&quot;Here I want to look into the dictionary to check if the element exists in it, 
                          if not print it and add to the dictionary at [entry_id] so we can skip it in next iteration
                          Don't know why but whenever a new item appears on the website it seems to edit every item in the dictionary instead of just editing the one
                          that wasnt there&quot;&quot;&quot;
                          print(
                              f&quot;{nickname} was killed by {killed_by} at {day}-{month}-{year} {hour}&quot;
                          )
                          self.records[
                              entry_id
                          ] = f&quot;{nickname} was killed by {killed_by} at {day}-{month}-{year} {hour}&quot;
          print(&quot;---&quot;)
          self.iter += 1
  
  
  ks = KillStatistics()
  if __name__ == &quot;__main__&quot;:
      while True:
          ks.parse_records()
          time.sleep(10)
  </code></pre>
  <p>The entry_id are always the same its 500 rows of data and they have id of 1,2,3...500 the newest is always 1. I know i could always check for 1 to get the newest but sometimes for example 10 players can die at the same time so i would like to check them all if they changed and only perform print on new ones.</p>
  <p><strong>Current output:</strong></p>
  <pre><code>Velerion was killed by Rat and Cave Rat at 27-12-2021 16:53
  Scrappy was killed by Cursed Queen at 27-12-2021 16:52
  Velerion was killed by Rat at 27-12-2021 16:28
  Velerion was killed by Rat at 27-12-2021 16:22
  Velerion was killed by Rat at 27-12-2021 16:21
  Velerion was killed by Rat at 27-12-2021 15:51
  Shade was killed by Tentacle Slayer at 27-12-2021 15:46
  Mr Yahoo was killed by Immortal Hunter at 27-12-2021 15:41
  Scrappy was killed by Witch Hunter at 27-12-2021 15:39
  Barbudo Arqueiro was killed by Seahorse at 27-12-2021 15:23
  Emperor Martino was killed by Dark Slayer at 27-12-2021 15:14
  Shade was killed by Tentacle Slayer at 27-12-2021 15:11
  Head Hunter was killed by Demon Blood Slayer at 27-12-2021 15:09
  </code></pre>
  <p><strong>Expected output:</strong></p>
  <pre><code>Velerion was killed by Rat and Cave Rat at 27-12-2021 16:53
  </code></pre>