title: Kubernets deployment doesn't finalize although our job is finished: 'the containers with unready status: [main]'
body: <p>My team and I, we are trying to deploy a BERT NLP model to production using Kubernets and Kubeflow.</p>
  <p>We almost got it. Everything was processed correctly and we've got our desired output as we can see in our log:</p>
  <p><a href="https://i.stack.imgur.com/7GWgb.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/7GWgb.jpg" alt="enter image description here" /></a></p>
  <p><strong>But the process simply doesn't finish</strong> and the last line in our DAG keeps with the 'loading' icon, as you can see here:</p>
  <p><a href="https://i.stack.imgur.com/Gv4S7.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Gv4S7.jpg" alt="enter image description here" /></a></p>
  <p>Eventually, checking on Kubernetes Engine, we realized an 'attention' icon to our job with the description <strong>'containers with unready status: [main]'</strong>.</p>
  <p><a href="https://i.stack.imgur.com/MWMWT.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/MWMWT.jpg" alt="enter image description here" /></a></p>
  <p>We can't find out what might be happening and we've being trying to find out for a whole week. I don't want to put everything we tried to not be more verbose on my post.</p>
  <p>If you want to check more about our deployment code:</p>
  <pre><code>import kfp
  from kfp.components import OutputPath, InputPath
  
  perform_query_op = kfp.components.create_component_from_func(
      perform_dataset_query,
      output_component_file='components/clf_perform_query.component.yaml',
      packages_to_install=[&quot;pandas&quot;, &quot;pyarrow&quot;, &quot;gcsfs&quot;, &quot;google-cloud-bigquery&quot;])
  treat_dataset_op = kfp.components.create_component_from_func(
      treatment_data,
      output_component_file=&quot;components/clf_treat_dataset.component.yaml&quot;,
      packages_to_install=[&quot;pandas&quot;, &quot;pyarrow&quot;, &quot;gcsfs&quot;, &quot;gensim&quot;])
  balance_categories_clf_op = kfp.components.create_component_from_func(
      balance_categories_clf,
      output_component_file=&quot;components/clf_balance_categories.component.yaml&quot;,
      packages_to_install=[&quot;pandas&quot;, &quot;pyarrow&quot;, &quot;gcsfs&quot;])
  create_train_test_split_op = kfp.components.create_component_from_func(
      create_train_test_split,
      output_component_file=&quot;components/clf_create_train_test_split.component.yaml&quot;,
      packages_to_install=[&quot;pandas&quot;, &quot;pyarrow&quot;, &quot;gcsfs&quot;, &quot;sklearn&quot;])
  train_model_op = kfp.components.create_component_from_func(
      train_transformers_model,
      output_component_file=&quot;components/clf_train_transformers_model.component.yaml&quot;,
      packages_to_install=[&quot;pandas&quot;, &quot;pyarrow&quot;, &quot;gcsfs&quot;, &quot;sklearn&quot;, &quot;torch&quot;, &quot;torchvision&quot;, &quot;tqdm&quot;, &quot;transformers&quot;, &quot;google-cloud-storage&quot;],
      base_image='pytorch/pytorch:1.9.0-cuda10.2-cudnn7-runtime')
  
  @kfp.dsl.pipeline()
  def train_classifier_pipeline():
      # Step 1: get data
      queried = perform_query_op()
  
      # Step 2: data treatment
      treated = treat_dataset_op(queried.output)
  
      # Step 3: Filtering and balancing categories
      balanced = balance_categories_clf_op(treated.output)
  
      # Step 4: Split datasets
      splits_A = create_train_test_split_op(balanced.outputs[&quot;brand_A&quot;], &quot;brand_A&quot;)
      splits_B = create_train_test_split_op(balanced.outputs[&quot;brand_B&quot;], &quot;brand_B&quot;)
      splits_C = create_train_test_split_op(balanced.outputs[&quot;brand_C&quot;], &quot;brand_C&quot;)
      splits_D = create_train_test_split_op(balanced.outputs[&quot;brand_D&quot;], &quot;brand_D&quot;)
  
      # Step 5: Train transformers model
      train_model_op(splits_acom.outputs[&quot;train&quot;], splits_A.outputs[&quot;test&quot;], &quot;brand_A&quot;)\
          .set_memory_request('20G')\
          .set_memory_limit('23G')\
          .set_gpu_limit('1').add_node_selector_constraint('cloud.google.com/gke-accelerator', 'nvidia-tesla-t4')
      train_model_op(splits_shop.outputs[&quot;train&quot;], splits_B.outputs[&quot;test&quot;], &quot;brand_B&quot;)\
          .set_memory_request('20G')\
          .set_memory_limit('23G')\
          .set_gpu_limit('1').add_node_selector_constraint('cloud.google.com/gke-accelerator', 'nvidia-tesla-t4')
      train_model_op(splits_suba.outputs[&quot;train&quot;], splits_C.outputs[&quot;test&quot;], &quot;brand_C&quot;)\
          .set_memory_request('20G')\
          .set_memory_limit('23G')\
          .set_gpu_limit('1').add_node_selector_constraint('cloud.google.com/gke-accelerator', 'nvidia-tesla-t4')
      train_model_op(splits_soub.outputs[&quot;train&quot;], splits_D.outputs[&quot;test&quot;], &quot;brand_D&quot;)\
          .set_memory_request('20G')\
          .set_memory_limit('23G')\
          .set_gpu_limit('1').add_node_selector_constraint('cloud.google.com/gke-accelerator', 'nvidia-tesla-t4')
  
  if __name__ == '__main__':
      from kfp.compiler.compiler import Compiler
      Compiler().compile(train_classifier_pipeline, &quot;components/train_classifier.pipeline.yaml&quot;)
      # Execute single run
      kfp_client = kfp.Client(host='my_path')  # PRD
      kfp_client.create_run_from_pipeline_func(train_classifier_pipeline, arguments={})
  </code></pre>
  <p>Any insights I will be grateful!</p>