title: moveToHDFS file not found in PySpark script
body: <p>I have a bit of a strange issue.</p>
  
  <p>When I run this command in terminal, it works and does indeed copy the file to the desired location.</p>
  
  <pre><code>hdfs dfs -copyFromLocal concsessions.csv /user/username/spark_exports/
  </code></pre>
  
  <p>But, when I run it as part of my script (below), it doesn't and I get this error - can anyone help me?</p>
  
  <p>I am not sure what I am doing wrong, definitely something!</p>
  
  <pre><code>OSError: [Errno 2] No such file or directory
  </code></pre>
  
  <p>Code:</p>
  
  <pre><code>from pyspark.sql import SparkSession
  from datetime import datetime
  
  #Set the date for the filename
  now = datetime.now()
  yday  = long(now.strftime('%s')) - 24*60*60
  
  spark = SparkSession\
  .builder\
  .appName('wap')\
  .master('yarn')\
  .enableHiveSupport()\
  .getOrCreate()
  
  import datetime
  import pyspark.sql.functions as F
  from pyspark.sql.functions import col
  
  #The below prints your results to your chosen destination (Hive, Stdout, CSV)
  
  print('data load starting...')
  
  cmd = '''select * from db.conc_sessions'''
  df1 = spark.sql(cmd)
  df1.printSchema()
  print('data ingested successfully')
  
  print('setting variables...')
  
  timestart= '2019-10-14 00:00:00'
  timeend= '2019-10-14 23:59:59'
  time_to_check = datetime.datetime.strptime(timestart, '%Y-%m-%d %H:%M:%S')
  
  iters = 0
  session = 0
  add = []
  
  print('begin iteration...')
  while iters &lt; 96:
  
      time_to_add = iters * 900
      time_to_checkx = time_to_check + datetime.timedelta(seconds=time_to_add)
      stringtime = time_to_checkx.strftime("%m/%d/%Y, %H:%M:%S")
  
      iters = iters + 1
  
      spark_date_format = "YYYY-MM-dd hh:mm:ss"
      df1 = df1.withColumn('start_timestamp', F.to_timestamp(df1.start_time, spark_date_format))
      df1 = df1.withColumn('end_timestamp', F.to_timestamp(df1.end_time, spark_date_format))
  
      filterx = df1.filter( (df1.start_time &lt; time_to_checkx) &amp; (df1.end_time &gt; time_to_checkx ))
  
      session = filterx.count()
      newrow = [stringtime, session]
      add.append(newrow)
  
  import pandas as pd
  output = pd.DataFrame.from_records(add)
  output.columns = ['time','count']
  output = output.groupby(['time'])[['count']].agg('sum').reset_index()
  output.to_csv('concsessions.csv', sep=',')
  
  #copy the CSV from the local server to HDFS
  import subprocess
  subprocess.call("hdfs dfs -copyFromLocal concsessions.csv /user/username/spark_exports/")
  </code></pre>