title: Number of rows affected by BigQuery query
body: <p>I run daily commands to insert new records into a BigQuery table, and would like to log how many records get inserted each day.</p>
  <p>I create a <a href="https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.job.QueryJob.html?highlight=query%20job#google.cloud.bigquery.job.QueryJob.query" rel="nofollow noreferrer"><code>QueryJob</code></a> object that contains a <code>SELECT</code> query, and a <code>destination</code> table.  I set the <code>write_disposition</code> to <code>WRITE_APPEND</code>, so that the new data is appended to the table.</p>
  <p>I've found two options that do something similar, but neither achieve what I'm looking for:</p>
  <ul>
  <li><code>query_job.num_dml_affected_rows</code>: This simply returns None, as the query doesn't use DML <code>INSERT</code>, but instead is appending to the destination table.</li>
  <li><code>query_job.result().total_rows</code>: This returns the total number of rows in the table, not the number of new rows.</li>
  </ul>
  <p>I can think of a variety of ways of achieving the desired result, but not sure what the best approach is:</p>
  <ul>
  <li>Change the query to a DML Insert - but this means creating dynamic SQL rather than just using the destination table is a python variable.</li>
  <li>Dump the result into a temporary table, count the rows, then append the data - but this just seems inefficient.</li>
  <li>Count the records before and after the query and log the delta - this can cause issues with running queries in parallel.</li>
  </ul>
  <p>Any suggestions no the best approach?</p>
  <hr />
  <p>Following up on muscat's answer, I think this wouldn't work when queries are run in parallel:</p>
  <pre><code>Get number of rows: 1000 rows
  Function to call queries in paralell:
  
   - Query 1 --&gt; Adds 100 rows --&gt; finishes 3rd --&gt; Counts 1200 rows
   - Query 2 --&gt; Adds 80 rows --&gt; finishes 2nd --&gt; Counts 1100 rows
   - Query 3 --&gt; Adds 20 rows --&gt; finishes 1st --&gt; Counts 1020 rows
  </code></pre>
  <p>Because there's no way to know which order these query will complete (as they are all called in parallel using using the <code>multiprocessing</code> library), I not sure how I can know how many rows each query added?</p>
  <hr />
  <h3>Update 2</h3>
  <p>Example code:</p>
  <pre><code>    ...
  
      # We compile a list of which datasets need to be loaded from
      brands = self._bq.select(f&quot;Select brand,  gaDataset From {self.BRAND_DATASET}.{self.BRAND_TABLE}&quot;)
      brands = list(brands.iterrows())
      _, brands = zip(*brands)
  
      # Define the function for parallel population
      def populate_fn(brand):
          return self._populate(brand, self.predicates)
  
      logging.info(&quot;Populating daily stats for brands in parallel&quot;)
      error = self._parallel_apply(populate_fn, brands)
      if error is not None:
          return error
  
  def _populate(self, brand, predicates):
      # We can't just call &lt;bq_load_data&gt; because we need to update the predicates for each brand
      predicates.add_predicate('gaDataset', brand['gaDataset'], operator=&quot;_&quot;)
      query_job = self._load_data(self.table_name, predicates=predicates)
      logging.info(f&quot;Started for {brand['gaDataset']}: {brand['brand']}&quot;)
  
      self._run_query_job(query_job)
      logging.info(f&quot;{brand['gaDataset']}: {brand['brand']} is now populated.&quot;)
  </code></pre>
  <p>The <code>_populate</code> function is run in parallel for each of the <em>brands</em>.</p>
  <p><code>predicates</code> is simply an object that handles how to modify the Jinja templated SQL, with some common parameters from the main object, and additionally some brand specific parameters.</p>
  <p><code>_load_data</code> is a function that actually loads the Jinja templated SQL with the appropriate parameters, and the constructs and returns a <code>QueryJob</code> object.</p>