title: Accessing files on Google Storage from a Google Cloud Python job
body: <p>I am loosely following a <a href="https://cloud.google.com/ai-platform/docs/getting-started-tensorflow-estimator" rel="nofollow noreferrer">tutorial</a> to train a TensorFlow estimator on Google Cloud AI Platform.</p>
  
  <p>I would like to access a directory that contains my training and evaluation data, and to this end I have copied my data files recursively to Google Storage like this:</p>
  
  <pre><code>gsutil cp -r data gs://name-of-my-bucket/data
  </code></pre>
  
  <p>This works fine, and <code>gsutil ls gs://name-of-my-bucket/data</code> correctly returns:</p>
  
  <pre><code>gs://name-of-my-bucket/data/test.json
  gs://name-of-my-bucket/data/test
  gs://name-of-my-bucket/data/train
  </code></pre>
  
  <p>However, calling <code>os.listdir(data_dir)</code> from a Python script raises a <code>FileNotFoundError</code> for any value of <code>data_dir</code> that I've tried so far, including <code>'data/'</code> and <code>'name-of-my-bucket/data/'</code>. Why?</p>
  
  <p>I know that my Python script is being executed from the directory <strike><code>/root/.local/lib/python3.7/site-packages/trainer/</code></strike> <code>/user_dir</code>.</p>
  
  <h1>Python code where the issue arises (edit)</h1>
  
  <p>Here is the code that precedes the line where the error arises, directly from the <code>__main__</code> section of my Python script:</p>
  
  <pre class="lang-py prettyprint-override"><code>PARSER = argparse.ArgumentParser()
  PARSER.add_argument('--job-dir', ...)
  PARSER.add_argument('--eval-steps', ...)
  PARSER.add_argument('--export-format', ...)
  
  ARGS = PARSER.parse_args()
  tf.logging.set_verbosity('INFO')
  os.environ['TF_CPP_MIN_LOG_LEVEL'] = str(tf.logging.__dict__['INFO'] / 10)
  
  HPARAMS = hparam.HParams(**ARGS.__dict__)
  </code></pre>
  
  <p>Here is the line of code where the error arises (first line of a separate function that gets invoked right after the lines of code I have reported above):</p>
  
  <pre class="lang-py prettyprint-override"><code>mug_dirs = [f for f in os.listdir(image_dir) if not f.startswith('.')]
  </code></pre>
  
  <h1>Logs (edit)</h1>
  
  <p>My logs for this job are a list of <em>infos</em> (plus 5 deprecation <em>warnings</em> related to TensorFlow), and then an <em>error</em> from the <code>master-replica-0</code> task:</p>
  
  <pre><code>Traceback (most recent call last): File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/root/.local/lib/python3.7/site-packages/trainer/final_task.py", line 114, in &lt;module&gt; train_model(HPARAMS) File "/root/.local/lib/python3.7/site-packages/trainer/final_task.py", line 55, in train_model (train_data, train_labels) = data.create_data_with_labels("data/train/") File "/root/.local/lib/python3.7/site-packages/trainer/data.py", line 13, in create_data_with_labels mug_dirs = [f for f in os.listdir(image_dir) if not f.startswith('.')] FileNotFoundError: [Errno 2] No such file or directory: 'data/train/'
  </code></pre>
  
  <p>... followed by another <em>error</em> from the same task (reporting non-zero exit status from my Python command), then two <em>infos</em> about clean-up, and finally an <em>error</em> from the <code>service</code> task:</p>
  
  <pre><code>The replica master 0 exited with a non-zero status of 1. Traceback (most recent call last): File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/root/.local/lib/python3.7/site-packages/trainer/final_task.py", line 114, in &lt;module&gt; train_model(HPARAMS) File "/root/.local/lib/python3.7/site-packages/trainer/final_task.py", line 55, in train_model (train_data, train_labels) = data.create_data_with_labels("data/train/") File "/root/.local/lib/python3.7/site-packages/trainer/data.py", line 13, in create_data_with_labels mug_dirs = [f for f in os.listdir(image_dir) if not f.startswith('.')] FileNotFoundError: [Errno 2] No such file or directory: 'data/train/' To find out more about why your job exited please check the logs: https://console.cloud.google.com/logs/viewer?project=1047296516162&amp;resource=ml_job%2Fjob_id%2Fml6_run_25&amp;advancedFilter=resource.type%3D%22ml_job%22%0Aresource.labels.job_id%3D%22ml6_run_25%22
  </code></pre>