title: How to submit spark batch jobs to HDInsight from Jupyter PySpark/Livy?
body: <p>I'm using an Azure HDInsight cluster and through the Jupyter interface included with HDI I can run interactive spark queries, but I was wondering how to run long running jobs. E.g. if during my interactive querying I realize I want to do some long running job that will take a few hours, is there way to run a command from PySpark itself, e.g. read data from path x, do some transformation, and save in path y?</p>
  <p>Currently if I just try to do that job inside the PySpark session itself and leave it running, Livy will eventually timeout and kill the job. Is there some command to submit the batch job and get an ID I can query later to get job status?</p>