title: bq.py Not Paging Results
body: <p>We're working on writing a wrapper for <a href="https://developers.google.com/bigquery/bq-command-line-tool" rel="nofollow noreferrer">bq.py</a> and are having some problems with result sets larger than 100k rows. It seems like in the past this has worked fine (we had related problems with <a href="https://stackoverflow.com/questions/17751944/google-bigquery-incomplete-query-replies-on-odd-attempts">Google BigQuery Incomplete Query Replies on Odd Attempts</a>). Perhaps I'm not understanding the limits explained on the <a href="https://developers.google.com/bigquery/docs/data" rel="nofollow noreferrer">doc page</a>?</p>
  
  <p>For instance:</p>
  
  <pre class="lang-sh prettyprint-override"><code>#!/bin/bash
  
  for i in `seq 99999 100002`;
  do
      bq query -q --nouse_cache --max_rows 99999999 "SELECT id, FROM [publicdata:samples.wikipedia] LIMIT $i" &gt; $i.txt
      j=$(cat $i.txt | wc -l)
      echo "Limit $i Returned $j Rows"
  done
  </code></pre>
  
  <p>Yields (note there are 4 lines of formatting):</p>
  
  <pre class="lang-none prettyprint-override"><code>Limit 99999 Returned   100003 Rows
  Limit 100000 Returned   100004 Rows
  Limit 100001 Returned   100004 Rows
  Limit 100002 Returned   100004 Rows
  </code></pre>
  
  <p>In our wrapper, we directly access the API:</p>
  
  <pre class="lang-python prettyprint-override"><code>while row_count &lt; total_rows:
      data = client.apiclient.tabledata().list(maxResults=total_rows - row_count,
                                                   pageToken=page_token,
                                                   **table_dict).execute()
  
      # If there are more results than will fit on a page, 
      # you will recieve a token for the next page
      page_token = data.get('pageToken', None)
  
      # How many rows are there across all pages?
      total_rows = min(total_rows, int(data['totalRows'])) # Changed to use get(data[rows],0)
      raw_page = data.get('rows', [])
  </code></pre>
  
  <p>We would expect to get a token in this case, but none is returned.</p>