title: Databricks job failing with no error message
body: <p>This is my first time posting a question. I have a very weird situation going on. So I am basically submitting a pyspark script via a databricks job with job and cluster parameters. Below is the cluster parameters I am using:</p>
  
  <pre><code>    'spark_version': '6.1.x-scala2.11',
      'node_type_id': 'Standard_DS12_v2',
      'autoscale': {'min_workers': 1, 'max_workers': 4},
      'cluster_log_conf': {
          'dbfs': {
              'destination': 'dbfs:/logs/{job_id}/logs'.format(job_id=job_id)
          }
      },
      'spark_env_vars': {
          'PYSPARK_PYTHON': '/databricks/python3/bin/python3',
      },...
  </code></pre>
  
  <p>The goal of the pyspark script is basically to train a scikit learn model and predict some test data. During training however, the predict function is also called to calculate some hyperparameters used to tune the model, which works fine. However, when the predict function is called when predicting the test data, it fails at scikit-learns' predict_proba method, with no error message...</p>
  
  <p>I have attempted a try catch around that part with no luck. The only feedback I get from the databricks logs are "Exception: The Databricks job did not complete successfully. The result state returned was FAILED".</p>
  
  <p>The only possible cause I have in mind currently is an out of memory error, but when that happened previously (with different cluster parameters) the logs actually return an out of memory error.</p>
  
  <p>Any and all help is greatly appreciated.</p>