title: Google ML Engine - Unable to log objective metric due to exception <HttpError 403>
body: <p>I am running a TensorFlow application on the Google ML Engine with hyper-parameter tuning and I've been running into some strange authentication issues.</p>
  
  <h2>My Data and Permissions Setup</h2>
  
  <p>My trainer code supports two ways of obtaining input data for my model:</p>
  
  <ol>
  <li>Getting a table from BigQuery.</li>
  <li>Reading from a <code>.csv</code> file.</li>
  </ol>
  
  <p>For my IAM permissions, I have two members set up:</p>
  
  <ol>
  <li><p>My user account:</p>
  
  <ul>
  <li>Assigned to the following IAM roles:
  
  <ol>
  <li>Project Owner (<code>roles/owner</code>)</li>
  <li>BigQuery Admin (<code>roles/bigquery.admin</code>)</li>
  </ol></li>
  <li>Credentials were created automatically when I used <code>gcloud auth application-default login</code></li>
  </ul></li>
  <li><p>A service account:</p>
  
  <ul>
  <li>Assigned to the following IAM roles:
  
  <ol>
  <li>BigQuery Admin (<code>roles/bigquery.admin</code>)</li>
  <li>Storage Admin (<code>roles/storage.admin</code>)</li>
  <li>PubSub Admin (<code>roles/pubsub.admin</code>)</li>
  </ol></li>
  <li>Credentials were downloaded to a <code>.json</code> file when I created it in the Google Cloud Platform interface.</li>
  </ul></li>
  </ol>
  
  <h2>The Problem</h2>
  
  <p>When I run my trainer code on the Google ML Engine using my user account credentials and reading from a <code>.csv</code> file, everything works fine.</p>
  
  <p>However, if I try to get my data from BigQuery, I get the following error:</p>
  
  <pre><code>    Forbidden: 403 Insufficient Permission (GET https://www.googleapis.com/bigquery/v2/projects/MY-PROJECT-ID/datasets/MY-DATASET-ID/tables/MY-TABLE-NAME)
  </code></pre>
  
  <p>This is the reason why I created a service account, but the service account has a separate set of issues. When using the service account, I am able to read from both a <code>.csv</code> file and from BigQuery, but in both cases, I get the following error at the end of each trial:</p>
  
  <pre><code>    Unable to log objective metric due to exception &lt;HttpError 403 when requesting https://pubsub.googleapis.com/v1/projects/MY-PROJECT-ID/topics/ml_MY-JOB-ID:publish?alt=json returned "User not authorized to perform this action."&gt;.
  </code></pre>
  
  <p>This doesn't cause the job to fail, but it prevents the objective metric from being recorded, so the hyper-parameter tuning does not provide any helpful output.</p>
  
  <h2>The Question</h2>
  
  <p>I'm not sure why I'm getting these permission errors when my IAM members are assigned to what I'm pretty sure are the correct roles.</p>
  
  <p>My trainer code works in every case when I run it locally (although PubSub is obviously not being used when running locally), so I'm fairly certain it's not a bug in the code.</p>
  
  <p>Any suggestions?</p>
  
  <h2>Notes</h2>
  
  <p>There was one point at which my service account was getting the same error as my user account when trying to access BigQuery. The solution I stumbled upon is a strange one. I decided to remove all roles from my service account and add them again, and this fixed the BigQuery permission issue for that member.</p>