where is final Output being stored of job/query executed on HDFS data.
lets suppose user has executed query on hive on HDFS data. And it return an output, where is this output being stored?
Is this HDFS or on Local file system?
Re: where is final Output being stored of job/query executed on HDFS data.
stored in a temporary scratch directory on HDFS and streamed to the client.
That’s correct, but the point is; what if its not a select statement rather some output to be used as an input of other job/query or be presented to some stake holders.
Could you please reply on my query?
All query results in Hive are stored in HDFS, not local filesystem.
Once the jobs end, the local directories of containers used in YARN do not persist, so it cannot be held in local filesystems.
Not certain what you exactly mean by 'used as an input of other job/query', but if you meant inter-stage data in a single query when two or more distinct YARN jobs are involved, then that stage result data too is on HDFS.
May I ask where the question/concern stems from?
As you mention, all hive result and inter-stage data stored in HDFS, that mean the result whether output or inter-stage data will be replicated by 3( or as per configuration)?
If all query(hive, impala etc) output stored in HDFS, then why we configure path data/1/impala/impalad for impala, yarn?
All of the answers here were for Hive, not Impala.
the life of the job.
they go on HDFS until automatically cleaned up by the query execution logic.