title: Scraping Real Time Visitors from Google Analytics
body: <p>I have a lot of sites and want to build a dashboard showing the number of real time visitors on each of them on a single page. (would anyone else want this?) Right now the only way to view this information is to open a new tab for each site.</p>
  <p>Google doesn't have a real-time API, so I'm wondering if it is possible to scrape this data. Eduardo Cereto found out that Google transfers the real-time data over the realtime/bind network request. Anyone more savvy have an idea of how I should start? Here's what I'm thinking:</p>
  <ol>
  <li><p>Figure out how to authenticate programmatically</p>
  </li>
  <li><p>Inspect all of the realtime/bind <em>requests</em> to see how they change. Does each request have a unique key? Where does that come from? Below is my breakdown of the request:</p>
  <p><a href="https://www.google.com/analytics/realtime/bind?VER=8" rel="nofollow noreferrer">https://www.google.com/analytics/realtime/bind?VER=8</a></p>
  <p>&amp;<code>key</code>= <strong>[What is this? Where does it come from? 21 character lowercase alphanumeric, stays the same each request]</strong></p>
  <p>&amp;<code>ds</code>= <strong>[What is this? Where does it come from? 21 character lowercase alphanumeric, stays the same each request]</strong></p>
  <p>&amp;<code>pageId</code>=rt-standard%2Frt-overview</p>
  <p>&amp;<code>q</code>=t%3A0%7C%3A1%3A0%3A%2Ct%3A11%7C%3A1%3A5%3A%2Cot%3A0%3A0%3A4%2Cot%3A0%3A0%3A3%2Ct%3A7%7C%3A1%3A10%3A6%3D%3DREFERRAL%3B%2Ct%3A10%7C%3A1%3A10%3A%2Ct%3A18%7C%3A1%3A10%3A%2Ct%3A4%7C5%7C2%7C%3A1%3A10%3A2!%3Dzz%3B%2C&amp;f</p>
  <p>The <code>q</code> variable URI decodes to this (what the?):
  t:0|:1:0:,t:11|:1:5:,ot:0:0:4,ot:0:0:3,t:7|:1:10:6==REFERRAL;,t:10|:1:10:,t:18|:1:10:,t:4|5|2|:1:10:2!=zz;,&amp;f</p>
  <p>&amp;<code>RID</code>=rpc</p>
  <p>&amp;<code>SID</code>= <strong>[What is this? Where does it come from? 16 character uppercase alphanumeric, stays the same each request]</strong></p>
  <p>&amp;<code>CI</code>=0</p>
  <p>&amp;<code>AID</code>= <strong>[What is this? Where does it come from? integer, starts at 1, increments weirdly to 150 and then 298]</strong></p>
  <p>&amp;<code>TYPE</code>=xmlhttp</p>
  <p>&amp;<code>zx</code>= <strong>[What is this? Where does it come from? 12 character lowercase alphanumeric, changes each request]</strong></p>
  <p>&amp;<code>t</code>=1</p>
  </li>
  <li><p>Inspect all of the realtime/bind <em>responses</em> to see how they change. How does the data come in? It looks like some altered JSON. How many times do I need to connect to get the data? Where is the active visitors on site number in there? Here is a dump of sample data:</p>
  </li>
  </ol>
  <pre class="lang-json prettyprint-override"><code>19
  [[151,[&quot;noop&quot;]
  ]
  ]
  388
  [[152,[&quot;rt&quot;,[{&quot;ot:0:0:4&quot;:{&quot;timeUnit&quot;:&quot;MINUTES&quot;,&quot;overTimeData&quot;:[{&quot;values&quot;:[49,53,52,40,42,55,49,41,51,52,47,42,62,82,76,71,81,66,81,86,71,66,65,65,55,51,53,73,71,81],&quot;name&quot;:&quot;Total&quot;}]},&quot;ot:0:0:3&quot;:{&quot;timeUnit&quot;:&quot;SECONDS&quot;,&quot;overTimeData&quot;:[{&quot;values&quot;:[0,1,1,1,1,0,1,0,1,1,1,0,2,0,2,2,1,0,0,0,0,0,2,1,1,2,1,2,0,5,1,0,2,1,1,1,2,0,2,1,0,5,1,1,2,0,0,0,0,0,0,0,0,0,1,1,0,3,2,0],&quot;name&quot;:&quot;Total&quot;}]}}]]]
  ]
  388
  [[153,[&quot;rt&quot;,[{&quot;ot:0:0:4&quot;:{&quot;timeUnit&quot;:&quot;MINUTES&quot;,&quot;overTimeData&quot;:[{&quot;values&quot;:[52,53,52,40,42,55,49,41,51,52,47,42,62,82,76,71,81,66,81,86,71,66,65,65,55,51,53,73,71,81],&quot;name&quot;:&quot;Total&quot;}]},&quot;ot:0:0:3&quot;:{&quot;timeUnit&quot;:&quot;SECONDS&quot;,&quot;overTimeData&quot;:[{&quot;values&quot;:[2,1,1,1,1,1,0,1,0,1,1,1,0,2,0,2,2,1,0,0,0,0,0,2,1,1,2,1,2,0,5,1,0,2,1,1,1,2,0,2,1,0,5,1,1,2,0,0,0,0,0,0,0,0,0,1,1,0,3,2],&quot;name&quot;:&quot;Total&quot;}]}}]]]
  ]
  388
  [[154,[&quot;rt&quot;,[{&quot;ot:0:0:4&quot;:{&quot;timeUnit&quot;:&quot;MINUTES&quot;,&quot;overTimeData&quot;:[{&quot;values&quot;:[53,53,52,40,42,55,49,41,51,52,47,42,62,82,76,71,81,66,81,86,71,66,65,65,55,51,53,73,71,81],&quot;name&quot;:&quot;Total&quot;}]},&quot;ot:0:0:3&quot;:{&quot;timeUnit&quot;:&quot;SECONDS&quot;,&quot;overTimeData&quot;:[{&quot;values&quot;:[0,3,1,1,1,1,1,0,1,0,1,1,1,0,2,0,2,2,1,0,0,0,0,0,2,1,1,2,1,2,0,5,1,0,2,1,1,1,2,0,2,1,0,5,1,1,2,0,0,0,0,0,0,0,0,0,1,1,0,3],&quot;name&quot;:&quot;Total&quot;}]}}]]]
  ]
  </code></pre>
  <p>Let me know if you can help with any of the items above!</p>
  <p><img src="https://i.stack.imgur.com/hF3p1.jpg" alt="enter image description here" /></p>