title: If I am using Dask-Jobqueue on a HPC, do I still need to use Dask-ML to run scikit-learn codes?
body: <p>If I am using Dask-Jobqueue on a High Performing Computer (HPC), do I still need to use Dask-ML (ie. <code>joblib.parallel_backend('dask'</code>) to run scikit-learn codes?</p>
  <p>Say I have the following code:</p>
  <pre><code>from dask_jobqueue import PBSCluster
  
  cluster = PBSCluster(cores=36,   
                       memory='100GB',   
                       project='P48500028',   
                       queue='premium',   
                       interface='ib0',
                       walltime='02:00:00')
  
  cluster.scale(100)  
                     
  from dask.distributed import Client
  client = Client(cluster)   
  
  
  from sklearn.datasets import make_classification
  from sklearn.svm import SVC
  from sklearn.model_selection import GridSearchCV
  
  X, y = make_classification(n_samples=2000, n_features=20, n_classes=2, random_state=0)
  
  param_grid = {&quot;C&quot;: [0.001, 0.01, 0.1, 0.5, 1.0, 2.0, 5.0, 10.0],
                &quot;kernel&quot;: ['rbf', 'poly', 'sigmoid'],
                &quot;shrinking&quot;: [True, False]}
  
  grid_search = GridSearchCV(SVC(gamma='auto', random_state=0, probability=True),
                             param_grid=param_grid,
                             return_train_score=False,
                             iid=True,
                             cv=3,
                             n_jobs=-1)
  
  
  import joblib
  
  with joblib.parallel_backend('dask'):
      grid_search.fit(X, y)
  </code></pre>
  <p>Since I am using Dask-Jobqueue on a HPC (ie. I am connected to an instance of the HPC), when I run my code would all of my code be distributed to a cluster already (since I have specified <code>cluster.scale(100)</code>)? If yes, then do I still need the last 3 lines of code above which uses Dask-ML? Or can my code be like this:</p>
  <pre><code>from dask_jobqueue import PBSCluster
  
  cluster = PBSCluster(cores=36,   
                       memory='100GB',   
                       project='P48500028',   
                       queue='premium',   
                       interface='ib0',
                       walltime='02:00:00')
  
  cluster.scale(100)  
                     
  from dask.distributed import Client
  client = Client(cluster)   
  
  
  from sklearn.datasets import make_classification
  from sklearn.svm import SVC
  from sklearn.model_selection import GridSearchCV
  
  X, y = make_classification(n_samples=2000, n_features=20, n_classes=2, random_state=0)
  
  param_grid = {&quot;C&quot;: [0.001, 0.01, 0.1, 0.5, 1.0, 2.0, 5.0, 10.0],
                &quot;kernel&quot;: ['rbf', 'poly', 'sigmoid'],
                &quot;shrinking&quot;: [True, False]}
  
  grid_search = GridSearchCV(SVC(gamma='auto', random_state=0, probability=True),
                             param_grid=param_grid,
                             return_train_score=False,
                             iid=True,
                             cv=3,
                             n_jobs=-1)
  
  grid_search.fit(X, y)
  </code></pre>
  <p>Will the last line of code above <code>grid_search.fit(X, y)</code> not run on any Dask cluster since I have removed <code>joblib.parallel_backend('dask')</code>? Or will it still run on a cluster since I have earlier on declared <code>cluster.scale(100)</code>?</p>
  <p>Many thanks in advance.</p>