title: MLflow Kubernetes Pod Deployment
body: <p>I'm attempting to create a kubernetes pod that will run MLflow tracker to store the mlflow artifacts in a designated s3 location. Below is what I'm attempting to deploy with</p>
  
  <p>Dockerfile:</p>
  
  <pre><code>FROM python:3.7.0
  
  RUN pip install mlflow==1.0.0
  RUN pip install boto3
  RUN pip install awscli --upgrade --user
  
  ENV AWS_MLFLOW_BUCKET aws_mlflow_bucket
  ENV AWS_ACCESS_KEY_ID aws_access_key_id
  ENV AWS_SECRET_ACCESS_KEY aws_secret_access_key
  
  COPY run.sh /
  
  ENTRYPOINT ["/run.sh"]
  
  # docker build -t seedjeffwan/mlflow-tracking-server:1.0.0 .
  # 1.0.0 is current mlflow version
  </code></pre>
  
  <p>run.sh:</p>
  
  <pre><code>#!/bin/sh
  
  set -e
  
  if [ -z $FILE_DIR ]; then
    echo &gt;&amp;2 "FILE_DIR must be set"
    exit 1
  fi
  
  if [ -z $AWS_MLFLOW_BUCKET ]; then
    echo &gt;&amp;2 "AWS_MLFLOW_BUCKET must be set"
    exit 1
  fi
  
  if [ -z $AWS_ACCESS_KEY_ID ]; then
    echo &gt;&amp;2 "AWS_ACCESS_KEY_ID must be set"
    exit 1
  fi
  
  if [ -z $AWS_SECRET_ACCESS_KEY ]; then
    echo &gt;&amp;2 "AWS_SECRET_ACCESS_KEY must be set"
    exit 1
  fi
  
  mkdir -p $FILE_DIR &amp;&amp; mlflow server \
      --backend-store-uri $FILE_DIR \
      --default-artifact-root s3://${AWS_MLFLOW_BUCKET} \
      --host 0.0.0.0 \
      --port 5000
  </code></pre>
  
  <p>mlflow.yaml:</p>
  
  <pre><code>---
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: mlflow-tracking-server
    namespace: default
  spec:
    selector:
      matchLabels:
        app: mlflow-tracking-server
    replicas: 1
    template:
      metadata:
        labels:
          app: mlflow-tracking-server
      spec:
        containers:
        - name: mlflow-tracking-server
          image: seedim/mlflow-tracker-service:v1
          ports:
          - containerPort: 5000
          env:
          # FILE_DIR can not be mount dir, MLFLOW need a empty dir but mount dir has lost+found
          - name: FILE_DIR
            value: /mnt/mlflow/manifest
          - name: AWS_MLFLOW_BUCKET
            value: &lt;aws_s3_bucket&gt;
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: aws-secret
                key: AWS_ACCESS_KEY_ID
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: aws-secret
                key: AWS_SECRET_ACCESS_KEY
          volumeMounts:
          - mountPath: /mnt/mlflow
            name: mlflow-manifest-storage
        volumes:
          - name: mlflow-manifest-storage
            persistentVolumeClaim:
              claimName: mlflow-manifest-pvc
  
  ---
  apiVersion: v1
  kind: Service
  metadata:
    name: mlflow-tracking-server
    namespace: default
    labels:
      app: mlflow-tracking-server
  spec:
    ports:
    - port: 5000
      protocol: TCP
    selector:
      app: mlflow-tracking-server
  
  ---
  kind: PersistentVolumeClaim
  apiVersion: v1
  metadata:
    name: mlflow-manifest-pvc
    namespace: default
  spec:
    storageClassName: gp2
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 2Gi
  </code></pre>
  
  <p>I am then building the docker image, saving it to the minikube environment, and then attempting to run the docker image on a kubernetes pod. </p>
  
  <p>When I try this, I get CrashLoopBackOff error for the image pod and 'pod has unbound immediate PersistentVolumeClaims' for the pod created with the yaml. </p>
  
  <p>I'm attempting to follow the information here (<a href="https://github.com/aws-samples/eks-kubeflow-workshop/blob/master/notebooks/07_Experiment_Tracking/07_02_MLFlow.ipynb" rel="nofollow noreferrer">https://github.com/aws-samples/eks-kubeflow-workshop/blob/master/notebooks/07_Experiment_Tracking/07_02_MLFlow.ipynb</a>). </p>
  
  <p>Is there anything noticeable that I'm doing wrong in this situation?</p>
  
  <p>Thank you </p>