title: Schedule a Job using the same PVC's as 1 other Pod in a StatefulSet
body: <p>In a Kubernetes cluster, I'd like to be able to schedule a Job (with a CronJob) that will mount the same Volumes as 1 Pod of a given StatefulSet. Which Pod that is is a run time decision, depending on the labels set on the Pod at the time of scheduling the Job.</p>
  
  <p>I guess many people will wonder why, therefore a description of what we're doing and trying to do:</p>
  
  <h1>Current setup</h1>
  
  <p>We have a StatefulSet which serves a PostgreSQL database. (one primary, multiple replica's)
  We want to be able to create a backup from one of the pods of the StatefulSet.</p>
  
  <p>For PostgreSQL we can already do backups over the network with <code>pg_basebackup</code>, however we are running multi-TB PostgreSQL databases, which means full streaming backups (with <code>pg_basebackup</code>) is not feasible.</p>
  
  <p>We currently use <code>pgBackRest</code> to backup the databases, which allows for incremental backups.</p>
  
  <p>As the incremental backup of <code>pgBackRest</code> requires access to the data volume and the WAL volume, we need to run the Backup Container on the same Kubernetes Node as the PostgreSQL instance, we currently even run it inside the same Pod in a separate Container.</p>
  
  <p>Inside the container, a small api wraps around <code>pgBackRest</code> and can be triggered by sending <code>POST</code> requests to the api, this triggering is currently done using CronJobs.</p>
  
  <h2>Downsides</h2>
  
  <ul>
  <li>Every PostgreSQL instance has multiple containers in the Pod, 1 to serve Postgres, 1 to serve a tiny wrapper around <code>pgBackRest</code></li>
  <li>Job Logs only show successful backup triggers, the actual backup logs are part of the Backup container</li>
  <li>The Pod that will run the backup may run on a relatively old configuration, changing backup configuration requires a rescheduling of the Pod, which may mean a fail over of the PostgreSQL primary.</li>
  </ul>
  
  <h1>Proposed setup</h1>
  
  <p>Have a CronJob schedule a Pod that has the same Volume's as 1 of the Pods of the StatefulSet. This will allow the backup to use these Volumes.</p>
  
  <p>However, which Volumes it needs is a run time decision: We may want to run the backup on the Volumes connected to the primary, or we may want to backup using the Volumes of a replica. The primary/replica may change at any moment, as auto-failover of the PostgreSQL primary is part of the solution.</p>
  
  <p>Currently, this is not possible, as I cannot in the CronJob spec find any way to use information from the k8s api.</p>
  
  <p>What does work, but is not very nice:</p>
  
  <ul>
  <li>Use a CronJob that schedules a Job</li>
  <li>This Job queries the k8s api and schedules <em>another</em> job</li>
  </ul>
  
  <p>For example, this is what we can do to have a job create another job using this run-time information:</p>
  
  <pre><code>apiVersion: batch/v1beta1
  kind: CronJob
  metadata:
    name: schedule-backup
  spec:
    schedule: "13 03 * * *"
    jobTemplate:
      spec:
        template:
          spec:
            containers:
            - name: backup-trigger
              image: bitnami/kubectl
              command:
              - sh
              - -c
              - |
                PRIMARYPOD=$(kubectl get pods -l cluster-name=&lt;NAME&gt;,role=master -o custom-columns=":metadata.name" --no-headers)
                kubectl apply -f - &lt;&lt;__JOB__
                  apiVersion: batch/v1
                  kind: Job
                  metadata:
                    name: test
                  spec:
                    volumes:
                      name: storage-volume
                      persistentVolumeClaim:
                        claimName:
                          data-volume-${PRIMARYPOD}
                    [...]
                __JOB__
  </code></pre>
  
  <p>The above may be best served by an Operator instead of using just a CronJob, but I'm wondering if anyone has a solution to the above.</p>
  
  <h2>Downsides</h2>
  
  <ul>
  <li>Job Logs only show successful backup triggers, the actual backup logs are part of <em>another</em> Job</li>
  <li>The Job requires permissions to schedule Pod's, requiring yet another role/rolebinding</li>
  <li>Using heredocs in Bash makes things harder to read/parse/understand </li>
  </ul>
  
  <h1>Summary</h1>
  
  <p>Long story, but these are the constraints we want to satisfy:</p>
  
  <ul>
  <li>Run a Backup of PostgreSQL database</li>
  <li>These are multi-TB databases</li>
  <li>Therefore, incremental backups are required</li>
  <li>Therefore, we need to mount the same PV of an already running Pod</li>
  <li>Therefore, we need to run a Pod (or container) on the same K8s Node as the PV</li>
  <li>We want to be able to express this in a CronJob spec, instead of having to do runtime kubernetes api calls</li>
  </ul>